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Abstract 

To further the understanding of ELL (under) achievement and broaden the current scope of OTL 
models, the primary focus of this study was to investigate process and content opportunities that are 
particularly relevant to improving ELL achievement with particular attention to the relationship 
between opportunities to acquire academic language and ELL achievement. The work reported here 
operationalized academic language within a systemic functional linguistics theory (also called 
"functional grammar"). This theory of language use provided teachers and students with a 
framework for analyzing language in a manner that both built on existing language knowledge (or 
schemas) and provided them with an accessible structure for examining more complicated language 
(and content) concept. Based on our literature review, previous findings, and new ELL-sensitive OTL 
model, we investigated the following research questions in this study: (a.) To what extent and in what 
ways are students being exposed to key OTL variables in classrooms? (b). What is the impact of 
academic language and other OTL indicators on ELLs' and non-ELLs' performance on LAP A? After 
an introduction and literature review, the remainder of this report addresses sources of data, 
description of instruments and procedures, and types of analyses selected for the study; findings 
related to the research questions; and concludes with a discussion based on the findings. One of the 
most important findings from this study points to the need for explicit instruction on academic 
language. The positive impact of functional grammar implementation on student outcomes also 
suggests that in order for ELLs to fully benefit from assessment-driven reform, teachers need the 
capacity to make the linguistic expectations clear to students by focusing on the linguistic elements 
that are characteristic of academic registers. 
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EXECUTIVE SUMMARY 



Past research has demonstrated that discrepancies in educational inputs 
across schools are linked to differences in academic achievement across groups of 
students (e.g., Aguirre-Munoz & Boscardin, forthcoming; Darling-Hammond, 1990, 
1994; Gross, 1993; Jackson, 1982; Kozol, 2000; Oakes, 1985). Also, the evidence of a 
relationship between student background characteristics including language 
proficiency and test scores is mounting. These trends point to the need for ongoing 
investigations of the differential impact of reform on different groups of students, 
especially English language learners (ELLs) with equal attention given to input (e.g., 
teacher training, student exposure to content, etc.); processes (e.g., instructional 
strategies and delivery format); and output (e.g., achievement, course taking 
patterns, etc.) (Bartman, 2002; Lee & Wong, 2004; Valencia, Valenzuela, Sloan, & 
Foley, 2004). Although there is a rapidly growing literature in this area, few studies 
have focused on factors that impact performance of ELLs — a significant proportion 
of students in public schools who may be most impacted by high-stakes 
accountability policies. Without systematically integrating factors that are 
instructionally relevant for ELLs, current opportunity to learn (OTL) models may be 
limited in expanding our knowledge of the antecedents of ELL learning and thus 
may not provide adequate guidance for developing reform policies that adequately 
address their instructional needs. 

To further the understanding of ELL (under) achievement and broaden the 
current scope of OTL models, the primary focus of this study was to investigate 
process and content opportunities that are particularly relevant to improving ELL 
achievement with particular attention to the relationship between opportunities to 
acquire academic language and ELL achievement. The primary research questions 
thus included: 

1. To what extent and in what ways are students being exposed to key OTL 
variables in classrooms? 

2. What is the impact of academic language and other OTL indicators on ELLs' 
and non-ELLs' performance on Language Arts Performance Assignment 
(LAPA)? 

In order to address the research questions, we used a mixed-methods 
approach by incorporating both quantitative and qualitative analyses. Additionally, 
we knew prior to our investigations that direct instruction on linguistic structures 



2 




that correspond to academic language would not be occurring in classrooms. 
Teachers are simply not provided with techniques for doing so in their credential 
courses (Wong-Fillmore, 2000). Given that one of our research interests was to 
examine the impact of ELL opportunity to learn academic language on student 
outcome, we needed to create an environment to study this explicitly. The work 
reported here operationalized academic language within a systemic functional 
linguistics theory (also called "functional grammar"). This theory of language use 
provided teachers and students with a framework for analyzing language in a 
manner that both built on existing language knowledge (or schemas) and provided 
them with an accessible structure for examining more complicated language (and 
content) concepts. This was achieved by deconstructing linguistic structures to 
highlight those that correspond to academic language. Therefore, we developed a 
four-day teacher training program on instructional strategies to incorporate 
functional grammar in classrooms with 2-day follow-up comprised of four modules 
to ensure some level of academic language instruction in classrooms. 

A summary of key findings that address these research questions are 
presented below: 

Ql: To what extent and in what ways are students being exposed to key OTL 
variables in classrooms? 

To address this question, we examined the instructional practice data from 
three sources: (a) the teacher OTL survey, (b) classroom observations, and (c) teacher 
interviews. While the OTL survey primarily measured the quantity of specific 
instructional practices aligned with OTL variables, the observation and interview 
data provided more information on the quality of those practices. The key findings 
are highlighted under each of the OTL variables. 

Content exposure: Academic language. Based on our survey, interview, and 
observation data, we found that teachers in general did not adequately expose 
students to functional grammar concepts. Although we did not find systematic 
differences between trained and comparison teachers in the level of explicit 
instruction on functional grammar concepts in the survey responses, we found 
qualitative differences in the level of functional grammar implementation between 
these two groups of teachers when we examined our interview and observation 
data. Comparison teachers tended to focus simply on content and ideas and on a 
broad and superficial level of writing instruction (e.g., an overall essay structure). 
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whereas most trained teachers described detailed writing lessons in their interviews, 
which included various prewriting activities that helped students develop ideas as 
well as instruction in academic language that met grade level expectations during 
the first draft and revision phases. 

Content exposure: ELA (English language arts) content coverage. On 

average, both trained and comparison teachers reported that they spent about 3 to 4 
weeks on various activities related to literary analysis. However, the types of 
instruction provided to students differed for the trained and comparison teachers. In 
regards to writing instruction, comparison teachers focused instruction on 
prewriting activities and some writing conventions, such as global essay structure 
and mechanics, whereas trained teachers indicated the use of a greater degree of 
instructional support in writing, including not only organization of ideas, but also 
expression of ideas in a coherent and authoritative manner. 

Access and development. This variable included ELL process strategies, 
second language acquisition, and delivery format. We found that teachers varied 
significantly in the amount of the various instructional strategies they utilized that 
specifically targeted ELLs. In general, the level of access and development strategies 
provided to students ranged from once per week to two or more times a day. Based 
on the interview data, we found that the majority of teachers were familiar with 
ELL-specific processes, and most provided at least one method consistently. 
However, most teachers provided mainly verbal scaffolding and direct instruction. 
Although teachers mentioned the use of procedural scaffolding, observations 
revealed that teachers tended to utilize whole-group guided instruction. Thus, the 
gradual move toward independent work was not observed. 

Further, teacher comments revealed a belief that verbal scaffolding in whole- 
group instruction, coupled with the use of some specialized techniques in the 
presentation of the content, such as graphic displays, provided sufficient access to 
the curriculum. Therefore, teachers tended not to provide small group activities that 
fostered negotiation of meaning; a process necessary for language learning. 

Feedback and assessment: LAPA preparation. According to our survey 
results, the amount of feedback teachers provided to the students varied 
significantly, ranging from about once a month to once or twice a week. Based on 
the interview and observation data, we found that the majority of trained teachers 
were providing students with meaningful and specific feedback, and conducting 
assessment of comprehension to a high degree. 
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Teacher experience and expertise. The average number of years teaching was 
about 10 years for the teachers participating in this study. In addition, the average 
number of years teaching English language arts (ELA) was about 6 years. However, 
about 50% of the teachers had less than 4 years of teaching experience in ELA. 

Q2: What is the impact of academic language and other OTL indicators on ELLs' 
and non-ELLs' performance on Language Arts Performance Assignment 
(LAPA)? 

Content exposure: Academic language. We found consistently across all four 
LAPA scores that students in classes with teachers who had high functional 
grammar implementation had higher performance on LAPA than students in the 
classrooms with low implementation of functional grammar concepts. Further, we 
found that the opportunity to learn functional grammar equally benefited both ELLs 
and non-ELLs. The level of functional grammar implementation was consistently the 
most important OTL variable for predicting student performance on all four scores, 
including the holistic and three functional grammar analytic dimensions. 

Content exposure: ELA content coverage. Contrary to our previous findings, 
in this study, the relationship between the levels of content coverage in literary 
analysis and student outcome was not statistically significant. For the most part, 
reading opportunities and practice were achieved using the adapted materials of the 
school with few opportunities for students to select their own reading materials. 
One possible explanation for this lack of association can be attributed to the fact that 
teachers without knowledge of academic language structures tended not to provide 
instruction on expository writing and focused instruction on pre-writing and the 
first draft of the text. Teachers with knowledge of academic language structures 
used this knowledge to develop focused lessons throughout the writing phases, 
particularly the revision phase of writing. Consequently, the effect of ELA content 
coverage may have been confounded by the level of explicit academic language 
instruction. 

Access and development: ELL process strategies and second language 
acquisition. The hierarchical linear model (HLM) analysis did not reveal a 
significant relationship between ELL process strategies and student achievement. 
However, the lack of significance may be due to several factors, including: (a) the 
language used in the survey items, (b) the lack of variability among teachers, and (c) 
difficulty differentiating the unique effect. 
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Feedback and assessment: LAPA preparation. HLM analysis did not reveal a 
significant relationship between LAPA preparation and student performance. 
Again, the lack of significance may be attributable to the focus on the superficial 
aspects of writing by most teachers and the fact that many teachers followed similar 
instructional patterns during the LAPA preparation period as they did during the 
rest of the year. 

Teacher experience and expertise. Although based on the HLM analyses, 
teacher expertise was not a significant factor in student performance on LAPA; 
qualitative data revealed key differences in teacher quality based on years of 
teaching experience. Level of experience was found to differentiate teachers in 
important ways. First, less experienced teachers harbored misconceptions about 
what constituted effective practices for ELLs. These teachers often described the use 
of practices that are less effective in supporting ELLs' linguistic needs, such as 
overuse of direct instruction or lack of thinking skills instruction. Second, teachers 
with greater knowledge of academic language structures were more adept in 
identifying specific instructional needs and developing instructional plans aimed at 
addressing those needs. These teachers in particular directed a greater degree of 
attention to the revision process and did so effectively. Additionally, latent class 
analyses (LCA) also suggested that there seemed to be a positive association 
between education level and the way teachers interpreted and answered the OTL 
questionnaire. 

In summary, while certain teaching practices corresponding to specific OTL 
were found to have a positive impact on student performance, specifically 
instruction in academic language, LAPA scores overall were fairly low compared to 
the proficiency criteria of the LAPA holistic rubric. This corresponds directly with 
the general low level and poor quality of OTL exposure for the majority of the 
students in this study. The results from this study underscore the need for 
systematic examination of OTL to monitor the quality of instruction. 

Summary of Conclusions and Recommendations 

Academic Language in English Language Arts 

One of the most important findings from this study points to the need for 
explicit instruction on academic language. The findings further suggest that without 
continual and linguistically supported access to the curriculum, ELLs may not 
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benefit from assessment-driven reform efforts; over time, unsupported access may 
lead to increases in achievement gaps. By focusing on these language schemas, this 
study also contributes to research addressing ELL instruction by revealing the 
importance of instructional practices designed to build on students' existing 
knowledge structures in a manner that does not rely on techniques that limit ELLs' 
opportunities to receive and produce grade-appropriate academic texts, such as text 
adaptation and the use of graphic displays. While scaffolding strategies that reduce 
the linguistic demand of content are necessary for ELLs with very low English 
proficiency, teachers need tools for gradually removing these scaffolds. If these 
scaffolds are not removed, ELLs may not develop the capacity to cope with rigorous 
content and in turn may never fully benefit from outcome-based reform efforts, even 
if such reforms directly address their instructional needs. 

Academic Language in Other Subject Matter Learning 

The findings reported here also suggest that the systemic functional linguistics 
approach to academic language instruction, combined with appropriate access and 
development strategies, offers a theoretically based framework to provide ELLs with 
significant access to rigorous curriculum not only in language arts but also in other 
content areas (e.g., Schleppegrell, 2002). 

The incorporation of functional grammar across the content areas can be 
achieved by highlighting the linguistic elements that correspond to each of the 
contextual variables that map onto key metalinguistic functions: field ( what's going 
on), tenor ( point of viezv), and mode ( text structure). Since this study demonstrated 
that students could be explicitly taught how to recognize these grammatical 
features, then it should be possible to train students to identify these features in 
other content texts to increase their understanding of how the text functions to order 
ideas and build knowledge (i.e., increase reading comprehension). 

Recommendations for Teacher Training in Academic Language 

The positive impact of functional grammar implementation on student 
outcomes also suggests that in order for ELLs to fully benefit from assessment- 
driven reform, teachers need the capacity to make the linguistic expectations clear to 
students by focusing on the linguistic elements that are characteristic of academic 
registers. The findings from this study also provide evidence of the necessity of 
careful consideration of the content and planning of training aimed at providing 
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such capacity. We found that if we only considered whether a teacher participated in 
the training or not, the impact of the training was insignificant after controlling for 
school factors. However, when we considered level of implementation, the impact 
was significant. 

Triangulation 

The observations of classroom practices revealed inconsistencies between the 
amount of OTL teachers reported in the survey instrument and what the research 
team directly observed in their classrooms. This pattern was particularly evident for 
the scales targeting academic language (ELL content coverage /functional grammar), 
ELL-specific process strategies, and feedback to students. This finding brings up the 
issue of the reliability and validity of using teacher surveys to gather information 
about instructional practices. 

Recommendations for Improving ELL-Sensitive OTL Instruments 

Based on the results of this study, we offer the following recommendations for 
improving OTL instruments that are sensitive to the instructional needs of ELLs: 

• Include more specific examples of academic language coverage. 

• Items targeting ELL process strategies should reflect the need for balance 
among whole group, group work, and independent work, and the need to 
include activities within these delivery formats for negotiation of meaning. 

• Items targeting scaffolding and adaptation of content should be designed 
around specific English language development (ELD) levels. 

• Include items incorporating types of ELL support strategies with content 
coverage. 

• Include items that target metacognitive strategies to develop reading 
comprehension. 

• Include more items that target comprehensible input. 

• Include items that target the extent of instructional time spent addressing 
management issues in the classroom. 

• Collect information from teacher logs and lesson plans. 

• Collect additional information through discourse analysis. 
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Introduction & Literature Review 



With the advent of high-stakes accountability, the inadequate distribution of 
educational resources and access to knowledge (Darling-Hammond, 1990, 1994; 
Gross, 1993; Kozol, 2000; Oakes, 1985) has resurfaced as a critical issue (Lee & Wong, 
2004). Some researchers argue that test-driven accountability may reduce 
achievement gaps because it serves as an external pressure for academic 
improvement; however, its effects on academic performance depend largely on 
school capacity and social support (Lee & Smith, 1999; Newmann, King, & Rigdon, 
1997). For example, the achievement gaps among racial and socioeconomic groups 
were found to be relatively small in states with more equitable distribution of school 
resources and classroom opportunity to learn (OTL) indicators (Wong & Lee, 1998). 
Further, this body of work reveals that an emphasis on performance measures does 
not guarantee increased equity in the distribution of student learning (e.g., Aguirre- 
Munoz & Boscardin, forthcoming; Lee & Wong, 2004; O'Day & Smith, 1993). 
Further, Lee and Wong found in a national study on the impact of test-driven 
accountability that state accountability reform efforts did not reduce the gap 
between racial or socioeconomic groups. 

These trends point to the need for ongoing investigations of the differential 
impact of reform on different groups of students with equal attention given to input 
(e.g., teacher training, student exposure to content, etc.), processes (e.g., instructional 
strategies and delivery format), and output (e.g., achievement, course taking 
patterns, etc.). Although there is a rapidly growing literature in this area, few studies 
have focused on factors that impact performance of English language learners 
(ELLs) — a significant proportion of students in public schools who may be most 
impacted by high-stakes accountability policies (Bartman, 2002; Lee & Wong, 2004; 
Valencia, Valenzuela, Sloan, & Foley 2004). Without systematically integrating 
factors that are instructionally relevant for ELLs, current OTL models may be 
limited in expanding our knowledge of the antecedents of ELL learning and thus 
may not provide adequate guidance for developing reform policies that adequately 
address their instructional needs. 

To further the understanding of ELL (under) achievement and broaden the 
current scope of OTL models, the primary focus of this study was to investigate 
process and content opportunities that are particularly relevant to improving ELL 
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achievement, with particular attention to the relationship between opportunities to 
acquire academic language and ELL achievement. 

Opportunity to Learn 

Past research has demonstrated that discrepancies in educational inputs across 
schools are linked to differences in academic achievement across groups of students 
(e.g., Aguirre-Munoz & Boscardin, forthcoming; Darling-Hammond, 1990, 1994; 
Gross, 1993; Jackson, 1982; Kozol, 2000; Oakes, 1985). Wang (1998), for example, 
investigated four dimensions of OTL: content coverage, content exposure, content 
emphasis, and quality of instructional delivery. Wang found that these OTL 
variables were significant predictors of both written and hands-on test scores. 
Specifically, content exposure was the most significant predictor of students' written 
test scores, whereas quality of instructional delivery was the most significant 
predictor of the hands-on test scores. These findings suggest that OTL should be 
investigated as a multidimensional construct. Other findings point to the need to 
examine achievement scores in light of both instructional strategies to which 
students are exposed to as well as student background factors such as language 
background, ethnicity, and gender that may be associated with performance (e.g., 
Abedi, Leon, & Mirocha, 2000; Saxe, Gearhart, & Seltzer, 1999). 

The evidence of a relationship between student background characteristics, 
including language proficiency and test scores, is mounting. Lor example, Abedi et 
al. (2000) found that students' language proficiency was negatively associated with 
performance on the National Assessment of Educational Progress (NAEP) in 
mathematics. Guiton and Oakes (1995) linked student background characteristics to 
content exposure. They found that teacher expectations were associated with the 
proportion of minority students and, regardless of initial achievement, those who 
were placed in lower level courses showed smaller gains over time than students of 
comparable achievement who were placed in higher level courses. 

These findings underscore the need to continue investigations that lead to more 
comprehensive OTL constructs. Such research can substantially increase 
understanding of factors that influence the achievement of underperforming groups 
of students such as ELLs. Expanding on Porter's (1991) OTL model and previous 
CRESST work (Aguirre-Munoz & Boscardin, forthcoming; Baker, Niemi, Herl, 
Aguirre-Munoz, Staley, & Linn, 1995; Boscardin, Aguirre-Munoz, Chinen, Leon, & 
Shin, 2004; Herman, Klein, & Abedi, 2000), we developed our new OTL framework 
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for the current study around inputs, processes, and outputs with the intent to 
measure the levels of ELL-sensitive OTL in classrooms and examine which OTL 
variables significantly impact student achievement. 

In Porter's OTL model, inputs include general teacher quality, resources, 
student background, and parent and community norms. Processes include the 
organizational characteristics of schooling, such as the quality of state standards, 
and the instructional characteristics of schooling such as the curriculum and 
teaching quality. Outputs include achievement, participation, and attitudes and 
aspirations. While the OTL framework utilized in this study was influenced by the 
work of Porter, our aim was to expand on the content and processes factors to 
include variables that are specifically relevant for ELLs, namely, exposure to, and 
learning of, academic language, as well as sheltered instruction techniques that have 
been shown to be effective for ELL learning in the content areas. Of particular 
concern in this study are OTL measures that are sensitive to opportunities that ELLs 
receive in terms of content and process strategies as well as additional experiences 
ELLs require as students who are not proficient in English, such as comprehensible 
input, explicit language instruction, and sufficient opportunities to produce the 
target language (i.e., English). Next, we introduce our theoretical framework to 
provide rationale and context for development of our ELL-sensitive OTL indicators. 

Conceptualizing Academic Language 

Our review of the literature on academic language revealed that most of this 
literature was not based on systematic classroom-based investigations of the impact 
of academic language on student achievement. Essentially, most of the recent 
literature emphasized the need to increase students' understanding of academic 
language but fell short in providing empirically based definitions for academic 
language. Despite the paucity of research in this area, a number of scholars have 
broken ground in conceptualizing the features of academic language as well as some 
general principles for its measurement. 

Cognitive Academic Language Proficiency 

Among the first scholars to address the need to develop academic language 
was Jim Cummins. Cummins' work (Cummins, 1979, 1984) was important in that it 
suggested that proficiency in conversational English is necessary but insufficient for 
academic success. He argued that in order to address the educational issues of ELLs, 
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a distinction should be made between conversational and academic aspects of 
language proficiency. His early work characterized these two aspects of proficiency 
as basic interpersonal communicative skills (BICS) and cognitive academic language 
proficiency (CALP). Cummins arrived at this theory of second language acquisition 
after analyzing language acquisition data from the Toronto Board of Education 
(Cummins, 1981). These data indicated a gap between the attainment of 
conversational language proficiency and the attainment of grade norms in academic 
aspects of the second language. By suggesting that the language of schooling is 
cognitively distinct from the language used in everyday settings, Cummins directed 
attention to the importance of academic language in explaining the 
underachievement of ELLs. That is, "The distinction highlighted the fact that 
educators' conflating of these aspects of proficiency was a major factor in the 
creation of academic difficulties for bilingual students" (Cummins, 2000, p. 58). This 
position contrasted with the dominant view of proficiency at the time: the idea of 
one unitary proficiency dimension which implies that all individual differences in 
language proficiency could be accounted for by just one underlying factor. 

Specifically, Cummins (1996, 2000) argued that a unitary factor was inadequate 
for explaining the underachievement of ELLs. He pointed out that language should 
be considered from at least two dimensions: the level of contextualization and 
cognitive demand. Further, Cummins characterized the academic registers of 
schooling as context-reduced and cognitively demanding and everyday language as 
context-embedded and cognitively undemanding. Essentially, Cummins pointed out 
that acquiring proficiency in context-reduced language that involves cognitively 
demanding tasks requires a longer time than acquiring context-embedded language 
involving cognitively undemanding tasks. While this work has made an important 
contribution to the field, Cummins's theory regarding academic language has also 
undergone a number of critiques. For example, Schleppegrell (2004b) points out that 
it is inaccurate to characterize academic registers as decontextualized (context- 
reduced) because all language is produced and used within a context. She notes that 
what differentiates academic registers from social registers are the grammatical 
elements that are used to provide the context. Others (e.g., Edelsky, 1990; MacSwan, 
2000) have criticized the notion of CALP as perpetuating the idea that ELLs' 
academic failure is attributed to their low cognitive /academic proficiency rather 
than to inadequate schooling and thereby promoting a "deficit theory" explanation 
for ELLs' underachievement. 
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These critiques notwithstanding, most scholars in this area agree that the 
language of schooling is distinct from language used outside the classroom, 
particularly in informal settings. Therefore, efforts to operationalize this construct 
with a high degree of specificity are beginning to emerge. 

Recent Definitions of Academic Language 

A recent corpus-based approach to defining academic language in K-12 
educational settings has been conducted by CRESST researchers (e.g., Bailey & 
Butler, 2002; Butler, Lord, Stevens, Borrego, & Bailey, 2004; Butler & Stevens, 1997; 
Stevens, Butler, & Castellon-Wellington, 2000). In this work, six sources of evidence 
that were analyzed extensively included: empirical studies revealing the 
relationship between ELL /English-only student performance on content 
assessments and their language demands; prerequisites implied in national and state 
content standards, as well as English as a second language standards; teacher 
expectations; and finally analysis of classroom exposure to academic language. 
These scholars aimed to provide a systematic account of the contexts of language 
use (e.g., classroom activities and materials) and specific types of language features 
and functions used within and across these contexts. 

While Butler and her colleagues' initial work highlighted general and 
discipline-based vocabulary and syntax, they recently developed a framework that 
organizes this information into language functions. A focus on language functions 
permits the examination of "the language students must understand and use to 
complete educational tasks" (Butler et al., 2004, p. 7). Examples of language 
functions include description, explanation, definition, and persuasion. Although 
others (e.g., O'Malley & Valdez-Pierce, 1996) have identified language functions and 
have proposed suggestions for integrating the use of linguistic structures (e.g., 
highlighting transitional words and phrases that are common to particular language 
functions) with content area instruction, this work lacks theoretical coherence 
(Christie, 1999; Schleppegrell, 2004). Without a clear framework, teachers are less 
likely to develop integrated instructional activities that develop students' skills in 
both producing and comprehending extended academic discourse (oral and written). 
Both skills in oral and written modes are necessary for successfully completing 
academic tasks, particularly in developing new understandings where very little or 
no prior knowledge exists (Gibbons, 2002). 
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The work reported here operationalizes academic language within a systemic 
functional linguistics theory. This theory of language use, discussed in greater detail 
below, provides teachers and students with a framework for analyzing language in a 
manner that both builds on existing language knowledge (or schemas) and provides 
them with an accessible structure for examining more complicated language (and 
content) concepts. This is achieved by deconstructing linguistic structures to 
highlight those that correspond to academic language. Before we discuss this theory, 
however, we present issues with how academic language has been characterized by 
other scholars in this area. 

Academic or "literate" language (as opposed to oral language) is typically 
described as decontextualized, explicit, and complex by many scholars (e.g., 
Cummins, 2000; Gumperz, Kaltman, & O'Connnor, 1984; Michaels & Cazden, 1986; 
Olson, 1977, 1980; Snow, 1983; Torrance & Olson, 1984). Many of these scholars have 
concluded that these features suggest a higher cognitive demand of academic 
language over language utilized in informal conversation. Functional grammarians 
argue that these descriptors are inadequate and inaccurate characterizations (e.g., 
Schleppegrell, 2004b). As mentioned above, to refer to academic texts as 
decontextualized suggests that "these texts are somehow outside of any particular 
context" (Schleppegrell, 2004b, p. 9). First, all language occurs in context, including 
the language of schooling. What makes academic language difficult for many 
students is the need to rely on a different set of linguistic resources than what 
students encounter outside of school. These differences stem from the fact that, 
unlike oral discourse patterns, academic texts do not depend on shared experiences 
(Gibbons, 2002; Mercer, 2004), and thus are governed by a different set of lexical and 
grammatical resources. Much of the meaning making achieved in social interaction 
is achieved from the shared experiences and knowledge between interlocutors, 
whereas written texts require different kinds of contextualizing features for 
understanding because it "realizes different situational contexts" (Schleppegrell, 
2004b, p. 9). 

Second, the notion of explicitness is also challenged because it is typically 
defined as lexicalization and associated with clarity (Schleppegrell, 2004b). 
However, studies have demonstrated that lexicalization does not necessarily make a 
text clear or unambiguous. Explicitness depends on the familiarity with expectations 
of a given discourse situation. That is, contextual assumptions are important to 
interpretation. For this reason, it is almost impossible to make written text fully 
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explicit (Sinclair, 1993). Romaine (1984 as cited in Schleppegrell, 2004b) has 
demonstrated that children are familiar with the linguistic features necessary for 
making text lexically explicit. Where individual differences arise is in "knowing 
what is assumed and what must be made explicit in a particular situation" 
(Schleppegrell, 2004b, p. 11). 

Third, according to Schleppegrell (2004b), describing academic language as 
complex is also inappropriate as it suggests that literate text is superior to other 
types of text. Halliday (1987, 1989) argues that it is more accurate to consider oral 
and written discourse as having different kinds of complexity. A number of scholars 
have demonstrated that both forms are complex for different reasons. Schleppegrell 
(2004b) summarized this distinction as follows: 

Written, school-based texts tend to be complex in their internal clause structure, while spoken 
interaction tends to be complex in the way clauses are chained and linkages are indicated from one 
part of a larger discourse to another, (p. 13) 

In other words, the resources for creating meaning are different for these two 
discourse forms. Systemic functional linguistics highlights the specific linguistic 
structures that are functional within a given context where they appear, regardless 
of whether they are oral or written. Using the systemic functional linguistic 
approach, we can provide teachers and students with insight into how to 
comprehend, analyze, and construct written academic texts that utilize various 
linguistic features in order to realize a range of contexts appropriate for meeting 
school expectations. In the following sections, we provide a more detailed 
description of systemic functional linguistic theory and also describe how academic 
language was conceptualized as an OTL indicator in our new ELL-sensitive OTL 
model. 

The Systemic Functional Linguistic Approach 

Systemic functional linguistics (also referred to as functional grammar) offers a 
framework for examining different systems of language that provide resources for 
creating meaning (Halliday, 1975, 1994). This linguistic theory views language as "a 
social process that contributes to the realization of different social contexts. . . [and]. . . 
identifies how grammatical structures realize social meanings and how the 
meanings construe different contexts" (Schleppegrell, 2004b, p. 45). Like other 
schools of functional linguists, systemic functional linguistic theorists view language 
as inseparable from meaning; meaning cannot be divorced from the language used 
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to convey it. Working from the premise that grammar is a resource for making 
meaning (rather than a set of discrete rules), functions of clausal structures are 
examined in relation to the total linguistic system, and explicit links are made to 
"contextual variables to show how the situational context is realized through 
linguistic choices" at the clausal level (Schleppegrell, 2004b, p. 45). In this way, 
clausal elements are critical in understanding how different grammatical systems 
systemically interact with one another in order to constitute academic registers. 

Functional linguistics also allow us to consider both the cognitive and 
sociological aspects of language development. With respect to the investigation of 
cognitive development. Painter (2000) characterizes the advantage of the study of 
language development from a functional linguistic perspective. She states: 

A study of language development from a [functional linguistics] perspective is a study of 
conceptual development. If language itself is theorized as a system for making meaning, including 
an ideational component which functions in the interpretation of reality, then in exploring 
development we are exploring the individual’s growing capacity to make sense of experience. 

This means that as we map children’s changing linguistic ‘meaning potential’ we simultaneously 
build up a picture of their knowledge and capacity to think using symbols [stress added]. And 
since children’s knowledge is created interactively in talk with others, an exploration of language 
development can also be an exploration of the process of teaching and learning [stress added]. 

(P-66) 

From this perspective, acquisition of academic language is viewed as a process 
of developing essential sociolinguistic competence required for accomplishing a 
variety of academic tasks in various contexts including the school setting. 
Essentially, the notion that writing for academic purposes is a social practice 
underscores the need for systematic investigations of what is conventionally 
regarded as appropriate academic language. 

Register 

Systemic functional linguistic approaches to language allow us to observe 
distinctive features of academic language, which are clearly different from those of 
informal spoken language. The notion of register within the systemic functional 
linguistic approach offers a framework that enables us to specify linguistic elements 
and processes that characterize academic discourse in contrast to informal oral 
discourse. Register refers to "the configuration of lexical and grammatical resources 
which realizes a particular set of meanings" (Schleppegrell, 2004b, pp. 45-46). 
Central in the account of register is that a constellation of lexical and grammatical 
elements, each of which has its own meaning and function, generates a situational 
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meaning that particularly pertains to a given context. From a Hallidayan functional 
linguistic perspective, register variations are explained in three dimensions of 
language (i.e., field, tenor, and mode) that reflect key meta-functions of language. 

Field. The field of discourse is associated with presentation of ideas, thus 
typically involving "content" words such as nominal groups (text participants), 
verbal groups (processes), and adverbial expressions (circumstances). These 
linguistic elements construe the ideational meaning by representing experience 
expressed in discourse. The ideational meaning is further elaborated through 
connectors and other resources in which relationships among experiential elements 
are realized. Following Schleppegrell, Achugar, and Oteiza (2004), we characterized 
this dimension as what is going on in the text when working with teachers, 
particularly focusing on the elements of grammar that help the reader /listener 
understand events and the actors in those events. 

The field variables of academic register across content areas have 
commonalities in the sense that experiential elements of language are logically and 
clause-internally linked through various connectors beyond conjunctions. As a 
dimension associated with knowledge display, the field of academic discourse is 
composed of ideational resources realized in complex nominal structures (i.e., noun 
groups) with specialized, technical, and abstract vocabulary. 

Tenor. The tenor of discourse is closely related to the speaker or writer's 
display of stance (i.e., judgment or interpretation) in the text. The premise is that the 
speaker or writer expresses his or her personal stance in consideration of the listener 
or reader. Thus, the display of stance involves various linguistic resources that 
create the interpersonal meaning. Such interpersonal choices include mood, 
modality, intonation cues (in spoken discourse), and lexical elements that carry an 
evaluative and attitudinal meaning. When working with teachers, this dimension 
was referred to as the point of view, the grammatical elements that are used to 
express an implicit or explicit point of view. 

The tenor variables of academic register reflect a convention of academic 
discourse. That is, personal opinions and stances should be presented in an 
authoritative and impersonal fashion. This necessitates the use of interpersonal 
resources including the declarative mood, modal verbs, and lexical choices that 
carry an implicit evaluative meaning rather than choices that resort to an emotional 
appeal (e.g., rhetorical questions) and explicit evaluative meaning (e.g., 
constructions such as "I think that" and "I believe that"). 
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Mode. The mode of discourse refers to the way that language is structured in a 
given social context in which it is used. The structure of a text reflects both linguistic 
and nonlinguistic aspects of the social context, such as availability of feedback 
between speaker and hearer or between writer and reader. Linguistic resources that 
construe the textual meaning include cohesive devices such as conjunctions and 
connectors, clause-combining strategies, and thematic organization. We 
characterized this dimension as textual structure when working with teachers to 
reflect the elements of grammar that realize the type and organization of text that 
serves a specific purpose. 

The mode of academic register is realized through various linguistic processes 
that contribute to structuring an academic text in expected ways. These linguistic 
processes are frequently associated with high lexical density due to intricate linkages 
of ideas between and within clauses. The examples of such linguistic processes 
include clause structuring through subordinate clauses and embedded clauses, 
nominalization, and various theme choices that mark an organizational structure 
(e.g., use of interpersonal themes and connectors in theme position). 

In the following section, we discuss previous studies of academic register 
grounded in systemic functional linguistics in greater detail, particularly focusing on 
how they informed our conceptualization of academic language as a measurable 
OTL indicator. Specific examples of linguistic analysis are provided for an in-depth 
examination of how academic register is realized through various linguistic 
resources. 

Linguistic Analysis on Academic Language 

The functional linguistic view of language adopted in this project is 
fundamentally different from that of traditional linguistics. Heavily based on the 
Chomskyan notion of language as a mental representation of abstract structures, 
traditional linguistics has treated grammar in isolation from other dimensions of 
language such as meaning. In contrast, functional linguistic theorists view language 
as inseparable from meaning as previously mentioned. Working from this premise, 
some educational linguists have identified linguistic features within the language 
metafunctions of mode, tenor and field that characterize academic language. 

First, these researchers define the clause as a "message carrier" rather than a 
grammatical unit and use it as a critical unit of linguistic analysis. Second, Halliday's 
notion of "theme" is also utilized to analyze language. Referring to all the 
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grammatical elements that come before the main verb of a clause, theme functions as 
the starting point of a message for the clause (i.e., what the clause is going to be 
about). The analysis of theme is treated as an important construct for understanding 
how different grammatical systems systemically interact with one another in order 
to constitute academic language, particularly in analyzing how texts are structured 
to achieve the intended meanings 

Another Hallidayan notion related to theme is "rheme," which can be generally 
defined as the rest of the grammatical elements that come after the theme of the 
clause. We incorporate this element for the analysis of clause-to-clause cohesion. 
Building cohesion at the paragraph level involves direct linkages of the theme of one 
clause to the rheme of the previous clause. Strategies for creating linkages between 
clauses include nominalization, as will be described below, as well as the elaboration 
of noun phrases through embedding and further incorporation of adverbial 
expressions, to name a few. Thus, both theme and rheme are important for 
examining mode within text. 

Other features of written academic discourse have been identified by various 
linguists. Christie (2002a), for example, characterized "abstractness" (i.e., use of 
abstract nouns); "technicality" (i.e., use of technical language); and "grammatical 
metaphor" (i.e., the presentation of information using incongruent, atypical 
expressions through the use of strategies such as nominalization) as features of 
advanced academic writing. These features span the three metafunctions of mode, 
field and tenor, depending on the context of analysis. In particular, the ability to 
manipulate lexical and grammatical resources has been pointed out as a crucial 
ability for academic writing in this line of studies because it plays an important role 
in conveying ideas and knowledge in logical, coherent, and authoritative ways. In 
her discussion of linguistic demands for academic performance, Schleppegrell (2001) 
emphasizes multiple functions of nominalization, which refers to "an expression as 
a noun or noun phrase of what would more congruently be presented as a verb" (p. 
443), i.e., creating a noun phrase from what can be presented as a verb. The text 
below illustrates how a noun phrase ("the rapid expansion of the western territory 
that created new settlements in Arizona and Texas") originally appeared as a verb 
phrase ("was expanded"). 

The western territory was expanded as a result of the treaty of Guadalupe-Hidalgo. The rapid 

expansion of the western territory that created new settlements in Arizona and Texas led to 

increased populations in the southwest. 
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The first function relates to the condensed presentations of complicated ideas 
or processes. That is, nominalization elaborated by embedded clauses and 
prepositional phrases increases lexical density, allowing a concrete, condensed 
presentation of an idea that would otherwise be expressed in a lengthy sentence or 
set of sentences. In the example above, the adjective ("rapid"); the prepositional 
phrase ("of the western territory"); and the embedded phrase ("that created new 
settlements in Arizona and Texas") create lexical density. Second, nominalization 
enhances smooth transitions at the clausal level. Effective transitions achieved by the 
deployment of nominalization in the theme position often convey information 
expressed in a previous clause. This discourse strategy enhances a smooth transition 
from one clause to another by creating an intricate linkage between clauses. In the 
example above, the words "expanded" and "expansion" create a clause-to-clause 
link. Lastly, nominalization creates an impersonal, generic context in contrast to 
pronouns, such as first person references that invoke a personal context. With regard 
to other means of grammatical metaphor, Christie (2002a) and Schleppegrell (2001, 
2003, 2004) mention that the choice of mood and modality contributes to a 
personally detached, less subjective mode of writing. They argue that the declarative 
mood is highly valued in academic writing in comparison with rhetorical questions 
or exclamatory challenges, which often resort to a personal, emotional appeal. The 
following three sentences convey the same general assertion, but they clearly differ 
in the way that the interpersonal context of the argument is established. 

1. Who would not think that the expansion of the western territory resulted 
from the treaty of Guadalupe-Hidalgo? 

2. I think that the expansion of the western territory resulted from the treaty of 
Guadalupe-Hidalgo. 

3. It must be the case that the western territory resulted from the treaty of 
Guadalupe-Hidalgo. 

Compared to sentence 1 in the form of a rhetorical question and sentence 2 that 
starts with the first-person reference ("I") in theme position, sentence 3 achieves a 
highly impersonal, objective context of argument through an impersonal theme 
choice ("It") followed by a modal verb ("must") that implicitly conveys the author's 
epistemic stance toward the proposition. 

Academic Writing 

The functional linguistic approach has also been applied in analyzing students' 
writing performance, such as description (Schleppegrell, 1998, 2003); narratives 
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(Christie, 1986), scientific essays (Christie, 1986; Schleppegrell, 2003); literary 
analysis; and opinionated texts (Christie, 1986, 2002a). Functional linguistic analyses 
of student writing reported in previous studies reveal that students often lack 
understanding of expected language use in performing given academic tasks. In a 
description task, students invoke a non-academic interpersonal context (i.e., situated 
and personal context) by deploying the past progressive tense and first person 
references (Schleppegrell, 1998). In narratives, young writers produce mere recounts 
of temporal events that lack a sense of crisis or complication typical of narratives. 
This is evidenced by the overuse of action verbs (and under use of mental verbs) and 
lack of variety in connector choices (Christie, 1986). Similar problems are also found 
in students' character studies and tasks that call for deep literary analysis. Christie 
(1986, 2002a) shows that often missing from student writing is their own 
interpretation of characters and events. This characteristic is revealed by a lack of 
verbs that represent the writer's attitude concerning the story (e.g., attitudinal verbs 
such as resented, detested, and admired); lack of connectors that signal 
interpretation (e.g., because, although, if); and minimal references made to 
characters other than the main character. These qualities all indicate these students' 
difficulty in producing the linguistic features within academic language that 
comprise the three metafunctions of field, mode and tenor, with mode and tenor 
being the most difficult for English learners. 

The results of a functional linguistic analysis of student writing reveal the need 
for explicit instruction of how lexical and grammatical resources are closely linked 
to the realization of particular genres of school-based writing (Schleppegrell, 2003). 
Unfortunately, contemporary grammar instruction is practiced in the most 
decontextualized form, that is, teaching discrete grammatical points rather than how 
different grammatical systems create meaning in concert with each other. Christie 
(1986) criticizes the contemporary literacy curriculum that focuses merely on 
"content," "ideas," and "knowledge" in a manner that is highly detached from 
linguistic features used in expressing them. This issue is significant for ELLs whose 
lack of linguistic resources for expressing their ideas is often confused with cognitive 
learning disabilities (Artiles & Ortiz, 2002; Valdes & Figueroa, 1994). Functional 
linguistic approaches to language allow us to illuminate how a certain genre of 
academic discourse is realized through a group of lexical and grammatical items 
that characterize it. By implementing a functional linguistic approach to writing 
instruction, teachers can provide more explicit instruction of genre-specific features 
of academic language to enhance reading comprehension and writing skills. 
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Furthermore, teachers can be empowered with an analytical tool for analyzing 
students' writing more holistically in a way that moves beyond identifying spelling 
and punctuation errors or use of technical or "descriptive" vocabulary. Under this 
premise, explicit instruction in academic language is a critical element to 
conceptualization of an ELL-sensitive OTL model. 

Opportunities for Access and Development of Content and Language 

As part of expanding on previous OTL framework to include more ELL- 
sensitive OTL indicators, in addition to inclusion of academic language in the OTL 
model, we also identified key instructional strategies that have been shown to be 
effective for ELLs. These strategies correspond to three general areas: (a) second 
language acquisition strategies to represent how the language of the content is 
addressed to increase comprehensibility (e.g., comprehensible input, pre-view key 
vocabulary, predict, summarize, etc.); (b) ELL process strategies (e.g., scaffolded, 
explicit, and individualized instruction, and enticement activities); and (c) the 
delivery format of instruction (e.g., whole-group discussions, collaborative group 
work, and independent work). Both the first and second categories are designed to 
provide ELLs with access to the curriculum and involve varying degrees of 
scaffolding. Elowever, they differ in that content presentation techniques within ELL 
process strategies are generally used to minimize the amount of linguistic input in 
scaffolded instruction while still presenting key ideas, concepts, and relationships 
between concepts and ideas, whereas second language acquisition strategies are 
intended to provide students with tools for either (a) making sense of the linguistic 
input they receive to increase comprehensibility for the learner or (b) producing 
linguistic output that meets expectations for academic discourse and is 
comprehensible to the listener or reader. These strategies are described in detail 
next. 

Second Language Acquisition Strategies 

Sheltered content instruction. One instructional approach that focuses on 
second language acquisition is sheltered content instruction (see Echevarria & 
Graves, 1998 for a detailed description and literature base). Introduced by Stephen 
Krashen in the early 1980s, further elaborated by Schifini (1985), this approach 
utilizes second language acquisition strategies in content area instruction. 
Specifically, this approach highlights the use of language and context to make 
information comprehensible. One underlying premise of sheltered instruction is the 
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use of comprehensible input, outlined by Krashen (1989), to refer to the idea that 
second language acquisition occurs if and only if language input is comprehensible 
to the language learner. That is, if language input contains forms and structures just 
beyond the learner's current level of proficiency in the language, then both 
comprehension and acquisition will occur. Similarly, language input that is well 
above or below the learner's current level of proficiency restricts development of the 
second language. 

The goal of the sheltered approach is to tailor instruction to students' linguistic 
needs such that they can comprehend the content of instruction to then maximize 
participation in content-area classes and foster second language development. If 
implemented successfully, sheltered instruction provides (a) access to the core 
curriculum, (b) English language development (ELD), and (c) opportunities for 
social integration into today's culturally and linguistically diverse classrooms 
(Echevarria & Graves, 1998). 

The key features unique to the sheltered approach include: (a) adapting 
academic content to the English proficiency level of the students; (b) using speech 
that makes information comprehensible to students; (c) providing sufficient wait 
time and identifying language objectives; (d) emphasizing language development 
with emphasis on key vocabulary and use of supplementary materials to a high 
degree; (e) clarifying ideas in the students' first language whenever possible; and (f) 
frequent integration of the students' background experiences (Echevarria & Graves, 
1998). 

While the sheltered approach is the most widely used method for instructing 
ELLs (Echevarria & Graves, 1998), its design centers on supporting "learning 
content," not on learning new registers per se. That is, sheltered instructional 
techniques support meaning making related to content concepts, but not the explicit 
articulation of that knowledge, and they often rely heavily on the visual context at 
the expense of written modes. Moreover, while this approach is appropriate for use 
with students with very low English proficiency or for providing initial support to 
ELLs for understanding highly complex ideas, it may lead to inadequate exposure to 
more abstract and impersonal texts (oral or written). Over time, this lack of exposure 
to the academic language that is required for school-based tasks may not serve well 
students with intermediate and higher levels of English proficiency in developing 
higher levels of academic English proficiency (Schleppegrell, 2004). For example, in 
assessment contexts, the reader /writer needs to refer to events that are not shared 
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by the evaluator. Thus, the vocabulary becomes more abstract, and the text more 
impersonal. A heavy reliance on visual displays can lead to inadequate exposure to 
written discourse (Gibbons, 2000), which we argue is critical for the development of 
academic language because, as discussed earlier, it contains linguistic structures that 
are not found in oral discourse. The impact of the lack of exposure and opportunities 
to develop academic language may be greater when policies for exiting students out 
of ETL status include scores on standardized English language arts tests, which 
contain and require a great amount of academic language. The challenge for teachers 
is to structure activities that capitalize on ELLs' comfort levels with oral discourse, 
using sheltered techniques and gradually move to activities where the focus is on 
written discourse (Cummins, 2000; Gibbons, 2000; Schleppegrell, 2004). Thus, while 
sheltered techniques are important, teachers also need to be adept at incorporating 
methods that focus on authentic academic registers. Other instructional strategies 
that may provide more access and opportunities to acquire academic language are 
outlined below. 

ELL Process Strategies 

Scaffolded instruction. Scaffolded instruction aims to assist students in 
accomplishing an instructional goal by adjusting the amount of support and 
assistance that is provided throughout various stages of instruction. Thus, 
scaffolding refers to the level of support and encouragement provided to students in 
order to perform at the next level of understanding to ensure progress in learning 
and development. As such, scaffolding instruction is critical and necessary to 
develop ELLs' growing language and content knowledge (Echevarria & Graves, 
1998). As the metaphor implies, this involves temporary structures provided by 
teachers, in oral and written modes to assist students to participate in and carry out 
complex processes before they are ready to do so alone (Peregoy & Boyle, 2005). 
Assisted participation offers practice and development of a skill as an integrated 
whole, rather than drill on smaller aspects of the skill one at a time. Once proficiency 
is achieved, the scaffold is no longer needed and may be dropped. 

Explicit instruction. Explicit (or direct) instruction has been shown to be 
effective for students at-risk for school failure (i.e., little prior knowledge, extensive 
histories of failure, or little familiarity with complex tasks) (Elowell, Fox, & 
Morehead, 1993). The purpose of explicit instruction is to provide the learner with a 
"pictorial" way of understanding information and to decompose a task into a series 
of successful routines (Mercer, Jordon, & Miller, 1996; Ross & Robinson, 1987). This 
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approach places an emphasis on systematic teacher-led presentations that provide 
students with structured techniques for acquiring information and structuring 
complex thoughts (Rosenshine, 1986; Simmons, Fuchs, Fuchs, Mathes, & Flodge, 

1995) . The three components of this instructional method include (a) teaching in 
small steps, (b) guiding students during initial practice, and (c) providing students 
with high levels of successful practice. The interactional patterns of this approach in 
language learning have been categorized as the initiation-response-evaluation (IRE) 
sequence (Sinclair & Coulthard, 1975). This approach, however, has been criticized 
as not providing students with opportunities for negotiation of meaning that 
promotes language development (Mehan, 1978; Schleppegrell & Simich-Dudgeon, 

1996) . Criticisms include limiting student responses to single words or short 
phrases; limiting the number of students that participate in the interaction; not 
providing instruction and practice in formulating linguistically elaborated 
responses; over-reliance on teacher interpretations; and finally, limiting students' 
opportunities to negotiate meaning or pursue their own questions or ideas (Mehan, 
1978; Schleppegrell & Simich-Dudgeon, 1996). 

Some researchers (e.g., Christie, 1998, 2002b; Wells, 1993), on the other hand, 
have shown that a direct approach to language learning can be effective in language 
learning if teachers are clear about its purpose and goals and use it at appropriate 
places in the lesson. These researchers (Christie, 1998, 2002b; Wells, 1993) have 
shown that direct instruction can be effective when students need to develop 
understandings of technical and grammatical language, especially if it is structured 
in a manner that students can reconstruct knowledge learned through experience in 
more formal academic language. This is typically achieved through a focus on new 
linguistic resources (Christie, 2002b; Gibbons, 2002). 

Delivery Format 

Opportunities for extended discourse. Opportunities for extended discourse 
can also serve an instructional purpose because the nature of communication 
between teacher and students and among students is critical for second language 
learning (Gibbons, 2002), as various scholars have found a positive relationship 
between oral language and achievement (see August & Flakuta, 1997, for a review of 
this literature). These studies generally show that students need to have 
opportunities to talk about the content in order to gain understanding of that 
content as well as control of the linguistic structures needed to convey such 
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understanding. The importance of adequate and meaningful opportunities for 
extended discourse is best summarized by Schleppegrell (2004) in the following: 

The learning of new registers, like learning a second language, requires appropriate input, 
opportunities for interaction and negotiation of meaning, and relevant focus on form that language 
takes in different settings and as it is used for different tasks. . . .The ability to use grammatical and 
lexical strategies of academic registers in writing and to recognize the meanings they make in 
the texts they read does not just come naturally in students' ordinary language 
development. Socialization into new registers depends on having interaction that is 
meaningful in the new contexts where those registers are functional. ... [Socialization 
into the new registers] requires meaningful and purposeful interactions with an 
interlocutor who is willing to pursue the meaning-making moves of the learner, (p. 153) 

Group or pair work. The use of collaborative groups is one method for 
providing opportunities for extended discourse. There are numerous studies citing 
the effectiveness of small collaborative group work. It is particularly important for 
improving achievement of ELLs. Specifically, effective group work has important 
advantages over whole-class work for second language learning (Gibbons, 2002; 
McGroarty, 1993) in several critical ways: 

1. Learners hear more language of a greater variety and more language directed 
toward them; group-work situations increase the input to the learner. 

2. Learners interact more with other speakers, and therefore their output is also 
increased. They tend to take more turns, and in the absence of the teacher, 
have more responsibility for clarifying their own meanings. In other words, it 
is the learners themselves who are doing the language learning work. 

3. What learners hear and what they learn is contextualized; language is heard 
and used in an appropriate context and used meaningfully for a particular 
purpose. 

4. There is likely to be considerable message redundancy. That is, similar ideas 
will be expressed in a variety of ways. Asking questions, exchanging 
information, and solving problems provide a context where words are 
repeated, ideas are rephrased, problems are restated, and meanings are 
refined. This redundancy supports comprehension because it gives learners 
several opportunities to hear a similar idea expressed in a number of ways. 

5. The need to get information or clarify meaning increases the opportunities for 
learners to ask questions that genuinely seek new information, and thus there 
is further input and practice in genuine communication as compared with 
whole-class contexts where it is much more usual for the teacher to ask the 
questions, and where students are often required to answer only the purposes 
of showing what they know. 
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Group work may have positive affective consequences: Learners who are 
intimidated by the whole-group structure are more confident in small groups and 
take more active roles in this context and thereby are more apt to produce the target 
language. 



New OTL Framework Including ELL-Sensitive OTL Indicators 

Based on this review of the literature, it was evident that previous OTL models 
lacked specific guidance in identifying opportunities that are particularly predictive 
of ELL achievement. Needed were constructs that capture the process and content 
factors that are particularly sensitive to the instructional needs of ELLs, such as 
access to, and guided instruction in, academic registers and other linguistic supports 
designed to provide students with greater access to the curriculum. The following 
framework (Figure 1) for investigating ELL opportunity to learn was developed to 
investigate the levels of OTL occurring in classrooms and the potential effects on 
student outcome. Newly added ELL-specific OTL indicators are highlighted in bold. 
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Content Exposure 

1. Content & Language Objectives 

• e.g., clear expectations 

2. Content Coverage 

• e. g v literary analysis, writing 
process 

3. Academic Language 

• e.g., development of student 
grammatical sensitivity and 
lexical knowledge of content 
as realized through mode, 
tenor and field 



Access & Development 

1. Delivery Format 

• e.g., whole, small, or pair group 
instruction, and extended 
discourse opportunities 

2. ELL Process Strategies 

• e.g., scaffolded, explicit, and 
individualized instruction, and 
enticement activities 

3. Second Language Acquisition 

Strategies 

• e.g., modified input, rate of 
speech, preview vocabulary, use 
of visuals and graphic displays 

4. Feedback & Assessment 

• e.g., specific and detailed 
responses to student 
performance 



Figure 1. ELL-sensitive OTL framework. 



Three general areas were investigated in the present study: (a) teacher expertise 
and experience, (b) content exposure, and (c) access and development. Each of these 
areas is expected to impact student achievement and is briefly described next. 

As one of the indicators of teaching quality, the Teacher Experience and 
Expertise variable contains three dimensions, including: (a) preparation and 
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training, b) number of years teaching, and (c) content knowledge. The first two 
variables reflect general teacher quality, which is part of the Input indicator category 
of Porter's model; and the last one is intended to reflect course-specific teacher 
quality, which is part of the Process indicator category of Porter's model. 

Teacher Experience refers to credential status, the number of years the teachers 
have been teaching in general, and the number of years teaching the content. These 
are teacher contextual variables that may impact pedagogical decisions and the level 
of instructional preparation a teacher undergoes before presenting the lesson. 
Teacher Expertise refers to the content and pedagogical knowledge teachers have 
acquired through pre-service and continued professional development activities. 

In this model, content knowledge is comprised of both knowledge of content 
and ideas of the discipline as well as the linguistic structures that comprise the 
discourse of a given discipline — the academic language that is necessary to 
accomplish discipline-based tasks. 

Content Exposure refers to the type and extent of the content topics covered in 
instruction, such as literary analysis, as well as the extent to which the teacher offers 
opportunities for the development of academic language. In addition, this variable 
refers to Content & Language Objectives, namely the clarity of teacher expectations 
as communicated to students regarding thier performance tasks. Of particular 
interest in this study is the degree to which teachers provide instruction that focuses 
directly on the language structures that comprise a given academic register. 
Expanding on previous CRESST work, content topics included both those related to 
English language arts (ELA), such as time spent on characterization as well as 
exposure to academic language pertinent to typical ELA content topics. 

Exposure to Academic Language content is defined in this context as coverage 
of functional grammar in the presentation of ELA content topics. In particular, the 
degree to which teachers provided instruction on genre-specific language structures 
related to field, tenor, and mode to enhance students' understanding of content 
topics as well as develop their skills in articulating their knowledge or producing 
targeted genres. 

Exposure to Content Coverage is defined as content topics covered in 
instruction that relate specifically to literary analysis (e.g., characterization, theme, 
etc.); reading (grade appropriate texts as well as reading text that reflects the genres 
targeted by the assessment); and writing activities. What distinguishes this variable 
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from academic language is that the topics that comprise this variable do not focus on 
explicit language instruction. 

Access and Development refers to instructional styles and activities that increase 
access to the curriculum (e.g., the use of visuals, comprehensible input, etc.) as well 
as foster development of deep understanding, including feedback to students 
through informal (e.g., interactions that encourage negotiation of meaning) or 
formal classroom assessments. Pedagogical knowledge includes both delivery 
formats that foster learning for all students as well as those that are particularly 
relevant for ELLs, such as grouping configurations that foster language learning. 

Delivery Format refers to various grouping configurations occurring in the 
classroom, whole- or small-group instruction as well as pair work. These delivery 
formats are either teacher or student led. ELL Process Strategies refers to specific 
delivery formats teachers use to meet the linguistic needs of ELLs, such as 
scaffolded, direct, and individualized instruction. Lastly, Second Language 
Acquisition Strategies refer to strategies borrowed from second language acquisition 
research that teachers use to reduce students' linguistic load while making the 
curricular content accessible. Some of these strategies include modified input, rate of 
speech, and use of visuals. These strategies differ from academic language 
instruction in that while they are designed to increase language acquisition by 
reducing the linguistic load (thus matching the students' language learning level), 
they therefore do not necessarily represent activities that expose ELLs to the grade 
level academic language they are expected to comprehend or produce in the 
classroom or assessment contexts. These strategies are particularly important for 
early language learning. 

Feedback and Assessment refers to both formal and informal interactions that 
provide teachers with information they can use to inform subsequent instruction, 
encourage negotiation of meaning and prepare students for assessments. Consistent 
with Porter's (1991) view, the type and quality of feedback and assessment reveals 
consistency between instructional goals and desired student outcomes. Being able to 
clearly relay to students what is being learned involves knowledge about the 
misconceptions students may hold related to the subject matter. Thus, feedback and 
assessment strategies involve those processes that provide teachers with this kind of 
knowledge about students in order to build on what students already know. For 
ELLs, this includes active negotiation of meaning, as students' utterances in English 
may not reflect what they actually know. Further, processes that involve negotiation 
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of meaning also have the potential for creating the context that fosters continued 
language learning. 



Research Questions 

Based on our literature review, previous findings, and new ELL-sensitive OTL 
model, we investigated the following research questions in this study: 

1. To what extent and in what ways are students being exposed to key OTL 
variables in classrooms? 

2. What is the impact of academic language and other OTL indicators on ELLs' 
and non-ELLs' performance on LAP A? 

The remainder of this report is organized as follows: the "Methods" section 
provides sources of data, description of instruments and procedures, and types of 
analyses selected for the study. The section entitled "Capturing Opportunity to 
Learn in Teacher Practice" reports on the findings related to the research question 1. 
"Results from HLM" presents the findings that address research question 2, and the 
final section presents the conclusions and discussion based on the findings. 



Methods 

In order to address the research questions, we used a mixed-methods approach 
by incorporating both quantitative and qualitative analyses. In this section, we 
provide a description of the research design, instrumentation, and analyses used to 
address our research questions. 

We knew prior to our investigations that direct instruction on linguistic 
structures that correspond to academic language would not be occurring in 
classrooms. Teachers are simply not provided with techniques for doing so in their 
credential courses (Wong-Pillmore & Snow, 2000). Given that one of our research 
interests was focused on examining the impact of ELL opportunity to learn 
academic language on student outcome, we needed to create an environment to 
study this explicitly. Therefore, we developed a 4-day teacher training program on 
instructional strategies to incorporate functional grammar in classrooms with a 2- 
day follow-up comprised of four modules to ensure some level of academic 
language instruction in classrooms. 
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Sample 



Teachers 

A sample of 32 language arts teachers from three urban middle schools in 
Southern California participated in the study. Out of 32 teachers, 21 teachers were 
assigned to receive professional training on a first-come first-served basis during the 
winter of 2004 (February or March) and additional follow-up training during the 
spring of 2004 (April or May). This group was over-sampled due to expected 
attrition rate and potential for low implementation. 

As shown in Table 1, more than half of the teachers participating in the study 
were from Wood Middle School, the largest of the three schools. The total years of 
teaching experience ranged from 1 to 27 years with an average of 10 years for both 
comparison and trained teachers (see Table 2). In general, comparison and trained 
teachers were very similar in terms of their teaching experience. Comparison group 
teachers did have a slightly higher average number of years teaching sheltered 
English than the trained teachers. 

Table 1 



Number of Teachers by School 





Frequency 


Valid percent 


Wood 


19 


59.4 


Casi 


4 


12.5 


Los Ninos 


9 


28.1 


Total 


32 


100.0 



Note. The names of schools are fictitious to retain anonymity. 



Table 2 

Background Information for Comparison and Trained Teachers 



Comparison Trained 



Variable Category N Mean SD N Mean SD 
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Total years of teaching 12 

Years at this school 12 

Years of teaching English language arts 10 
Years of teaching sheltered English 9 

Exact No. English/ LA courses - und 9 

Exact No. English/LA courses - grad 12 

No. courses - sheltered /SDAIE 11 

No. courses - ESL 12 



10.04 


7.34 


20 


9.55 


8.58 


4.63 


4.04 


20 


4.50 


4.12 


5.05 


5.35 


18 


7.78 


6.46 


5.33 


5.32 


15 


2.27 


2.25 


4.11 


2.80 


17 


3.65 


3.24 


2.25 


2.80 


14 


2.93 


3.05 


2.91 


1.58 


18 


2.22 


1.63 


2.08 


1.88 


17 


2.29 


1.61 



Although we only recruited language arts teachers, the majority of the teachers 
did not major in English or language arts as undergraduates (see Table 3). Twenty- 
four teachers did have teaching credentials. Only 20% of the trained teachers and 
33% of the comparison teachers did not hold teaching credentials. Seven out of 32 
teachers indicated that they either already had an advanced degree or were 
currently enrolled in a graduate program (see Table 3). 



Table 3 

Education Level 





Yes 


No 


Undergraduate major in English/ language arts 


7(21.9) 


25 (78.1) 


MA - Eng/ literature 


2 (6.3) 


30 (93.8) 


Currently enrolled in MA program 


5 (15.6) 


27 (84.4) 


Advanced degree (Ed.D., Ph.D., etc.) 


2 (6.7) 


27 (90.0) 



Students 

A total of 1,646 middle school students enrolled in language arts classes 
completed the Language Arts Performance Assignment (LAP A) at the end of spring 
2004. Table 4 presents the number and percentage of students by key background 
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variables (i.e., grade level, gender, language proficiency, and ethnicity). Students 
with California English Language Development Test (CELDT) scores and a 
designated English Language Development (ELD) level were classified as ELLs in 
this study. In general, the background characteristics of students from comparison 
and trained groups were very similar. The proportion of ELL students was slightly 
higher in the trained group as compared to our comparison group. About half of the 
students in our study were designated as ELLs. Also, most of our students were 
identified as Hispanic. 
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Table 4 

Proportion of Students by Key Background Variables 



Variable category 


Comparison 
N % 


N 


Trained 

O/ 

/o 


Gender 




Lemale 


327 


56 


372 


53 




Male 


260 


44 


326 


47 


ELL status 




Non-ELL 

students 


388 


57 


397 


43 




ELL Students 


290 


43 


531 


57 


Ethnicity 




Hispanic 


568 


94 


651 


86 




White 


7 


1 


27 


4 




Asian 


0 


0 


52 


7 




Other 


29 


5 


24 


3 




Missing 


74 




174 




Total 




678 




928 





Procedure and Instruments 



Teacher Training 

For the teacher training we developed a teacher resource binder which included four 
modules. On each day of the 4 days of training, a different module was introduced. 
Within each module, several whole- and small-group activities (including role-play), 
designed to provide teachers with practice in the application of the concepts or 
strategies learned, were integrated into the presentation of the material. The first 
two modules targeted the concepts of the functional linguistic approach, and the 
third module addressed instructional strategies, including language analysis, 
writing revision lessons, the instructional conversation, and the readers' and writers' 
workshop. Module four, presented during the final day of the training was 
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dedicated to the analysis of student writing, and the collaborative development of 
lessons using functional linguistic concepts. The following provides a description of 
each of the modules. 

Module 1 . Module 1 introduced three main linguistic categories for analyzing 
written discourse (i.e., how language presents information in the text: field, how 
language builds the text structure: mode, and how language conveys the writer's 
point of view in the text: tenor) as well as the fundamental functional grammatical 
concepts including participants, processes, and theme /rheme. Because these 
concepts are easily conflated with traditional linguistic concepts, such as noun 
phrases, verbs, and subject/predicates, this module particularly emphasized the 
usefulness of using functional grammatical concepts as an effective tool for 
examining language in the context where it is used. Examples of analyzed written 
texts were provided to teachers in order to help them compare a functional linguistic 
approach with a traditional linguistic approach and ultimately see the analytical 
power that a functional grammatical approach can provide. However, this module 
did not completely discard traditional linguistic concepts. For instance, traditional 
linguistic definitions of phrases, clauses and sentences, with which teachers are 
already somewhat familiar, were used as a basis for developing their knowledge of 
functional grammar. 

Module 1 also touched on how to conduct theme analysis as a means for 
evaluating written discourse, particularly in terms of the organization of the text at 
the paragraph level and the development of an interpersonal context. 

Module 2. Module 2 started with a review of the linguistic and cognitive 
demands for written responses to literature suggested by the California Academic 
Standards and Framework for English language arts. A comprehensive functional 
linguistics approach to text analysis of written character studies followed, focusing 
on identifying specific linguistic features that correspond to the suggested cognitive 
and linguistic demands of the genre. The focus on the character study was important 
because it was the type of task that was used as the outcome variable in this study. 

The presentation of text analysis followed the three general functional 
grammar categories introduced in Module 1 (field, tenor and mode). The first sub- 
section concerned the presentation of participants and processes and emphasized 
two things: (a) a well-written character study includes various types of processes, 
which encompass mental, attitudinal, and feeling processes that reflect the writer's 
evaluation and interpretation of the story, as well as action processes used in the 
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retell of the story; and (b) well-balanced references to characters other than the main 
character, as this balance is critical for depicting the main character's qualities as 
typically revealed in his or her interaction with others in the story. The second sub- 
section addressed clausal- and paragraph-level cohesion, in particular functions of 
various transitional expressions in relation to a character study (i.e., what functions 
transitional expressions perform in order to achieve cohesion, particularly in the 
context of a character study, for example those that mark time order, order of 
importance, compare and contrast, and cause and effect). The third sub-section 
discussed a number of techniques for expressing the writer's point of view 
implicitly, as this is characteristic of academic language. The techniques discussed in 
this section included how to choose words that intrinsically contain evaluative and 
affective content. Many examples excerpted from well-written character studies 
were also presented so that teachers gained insight into what they should strive for 
in student performance. 

Module 2 also discussed linguistics elements that are consistent across other 
academic genres. Not only did this part of the module reiterate features of well- 
written character studies as a genre of academic writing in more general terms, but 
also introduced features of academic writing that are more broadly applicable to any 
genre of academic writing. Newly introduced in this section were (a) thematic 
progression analysis adapted from Mauranen (1996) which allows us to see patterns 
in the development of argumentation flow typical of English academic texts; and (b) 
types of grammatical metaphor such as nominalization, which are linguistic 
strategies that differentiate academic language from informal spoken language. 

Module 3. Module 3 presented strategies for developing ELLs' skills in literary 
analysis and strategies for teachers to incorporate functional grammar in their 
writing instruction. The first section of Module 3 focused on analyzing both "model" 
and student text, and on using this information to elaborate on the revision stage of 
the writing process. Teachers and presenters together examined published text, such 
as students' textbooks, articles from newspapers, novels, non-fiction magazines, and 
student writing to analyze the patterns of linguistic choices authors use to realize 
meaning. Together they discussed how students can utilize this process to develop a 
repertoire of linguistic writing patterns to use in their own writing. 

Teachers then worked on developing writing revision lessons that respond to 
their analysis of model and student writing (Greenleaf & Freedman, 1993). They 
used a framework based on the work of Greenleaf and Freedman which 
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incorporates a three-step process, including (a) orientation, where the teacher 
introduces a particular student text to the class, its primary problem, and the 
revision process that will be used to improve it; (b) problem solving, where students 
and the teacher work together to revise the text; and (c) connection, where the 
teacher reinforces the transference of skills to students and their applicability in 
independent writing. The lessons, using a functional grammar approach, respond to 
lexical and grammatical weaknesses found in student writing. In the second section 
of Module 3, the instructional conversation (IC) approach was presented as a 
strategy for improving students' ability to conduct literary analysis. It is a model of 
interaction that uses a small-group format (5-7 students) to create opportunities for 
students to engage in thoughtful, reflective, sometimes provocative discussions 
about ideas, texts, and concepts (Genesee, Lindholm-Leary, Saunders, & Christian, 
2004; Goldenberg, 1993; Saunders & Goldenberg, 1999). The goals of the IC include: 

1. Provide a forum for developing new understanding and constructing 
meaning from the text. 

2. Improve the language skills and comfort levels of ELLs during which they 
can think, reflect, express ideas, and argue positions as they develop new 
understandings around a text. 

3. Develop higher-level cognitive skills, rather than factual recall. 

Finally, the readers '/writers' workshop (R/WW) was presented at the end of 
Module 3 to provide teachers with an overall instructional framework for 
implementing functional grammar. In the reading workshop, students had time to 
read in class, choices of books or other materials, access to books and materials, and 
opportunities for interaction. With a focus on student independence gained through 
utilization of the writing process, the writing workshop provided students with 
routines and linguistic and stylistic resources available to real world authors 
(Graves, 1983). 

The rationale for including these approaches in the training was to provide 
teachers with instructional processes that support ELL development of literary 
critique and academic writing (Allen, 1995). 

Module 4. In order to deepen teachers' knowledge of student writing patterns. 
Module 4 presented prevailing linguistic features of a character study found in 
English language learners' writing samples and identified specific areas of 
instructional support. In this module, teachers further developed plans for targeting 
ELL writing development. Specific areas of instructional need were discussed on the 
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basis of the following features of ELLs' character studies: (a) minimal references to 
characters other than the main character; (b) lack of a vocabulary repertoire that can 
be used for implicit expression of personal opinions and stance; (c) lack of cohesion 
as evidenced by monotonous theme choices (e.g., overuse of pronouns in theme 
position, and lack of adverbial expressions in theme position); (d) overuse of easy 
connectors such as "and," "so," and "because"; (e) lack of nominalization and 
expanded noun phrases that results in low lexical density; (f) failure to build an 
impersonal interpersonal context as exemplified by the use of oral discourse markers 
(i.e., well), and first- and second-person references; and (g) general lack of 
evaluation as manifested by the overuse of action processes. 

In addition to the modules, the training materials also included a selection of 
articles and book chapters pertaining to functional linguistics, writing instruction, 
ELL literacy development, and effective ELL strategies. While these readings came 
from a variety of sources, we attempted to include those that were particularly 
"reader-friendly" and that did not rely on technical jargon. Some selections were 
assigned as homework, with the first 30 minutes of each morning spent discussing 
the implications and issues addressed in each. The remaining selections were 
provided to teachers as additional resources if they felt a need or desire to delve 
deeper into the concepts and issues addressed in the institute beyond what was 
addressed in the week-long training. Evaluation of training effectiveness was 
conducted at the completion of the training. More information on training 
effectiveness is provided in Appendix A. 

Student Assessment 

The primary outcome measure for the study was a curriculum-embedded 
performance assessment designed to assess student understanding and skills in 
language arts. The Language Arts Performance Assignment (LAP A) was modeled 
after previous CRESST work and has undergone validation studies (Niemi, 
Sylvester, & Baker, 1998). In this assessment, students were asked to select a literary 
piece that contains a heroic character and describe the qualities of that character in 
writing, citing detailed information from the literary work. Among the 
characteristics they could write about were: physical and personality traits, thoughts 
and motivations, and relationships with other characters. Thus, students were 
expected to analyze the story beyond the surface features of the plot and support all 
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assertions about the text with accurate and supporting citations (See Appendix B for 
the LAP A writing prompt.) 

Students were also expected to go through the stages of the writing process in 
5-10 hours of class time over the course of 1-2 weeks with support from the teacher 
in the form of mini-lessons. If groups of students had difficulty with elements of the 
writing assignment, the teacher was allowed to provide a short 15- to 20-minute 
lesson to help them get through the assignment. Assistance, however, did not 
include direct feedback, such as editorial suggestions or the teacher's interpretations 
of the text. 

Scoring Student Work 

Rubric Development 

In addition to the holistic scoring rubric (see Appendix C) that was developed 
and validated in previous CRESST work (Boscardin et al., 2004), three additional 
dimensions evaluated on a 3-point scoring rubric were developed to evaluate 
student writing for this study. These dimensions evaluate students' performance on 
the LAPA based on the linguistic dimensions identified by the systemic functional 
linguistics approach utilized to operationalize academic language as noted above. 
While the existing CRESST rubric utilizes such holistic evaluation criteria as the 
clarity of character description, text-based support of argumentation, logical 
organization of ideas, and appropriate use of grammar, the systemic functional 
linguistics scoring rubric (heretofore referred to as "functional grammar") 
specifically focuses on assessing students' linguistic command of grammatical 
structures that are directly related to the written response to literature genre in 
general and specifically to characterization. 

The functional grammar dimensions aim to assess students' skills for 
effectively making use of the linguistic choices essential for successful 
characterization. The three linguistic dimensions are identified and classified 
according to their functions and presented in Table 5 below. 
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Table 5 



Description of Dimensions 





Language 


Function 


Dimension 1: Noun phrases 


Expanded noun phrases 


Describe persons, animals, 
things, and concepts 


Dimension 2: Lexical density 


Adverbial expressions 


Provide circumstantial 
expressions 


Dimension 3: Characters and references 


Tracking of participants 


Achieve well-balanced 
participants 



The assessment of students' linguistic command of each linguistic dimension 
was based on the following evaluation criteria. For Dimension 1, noun phrases were 
examined with regard to the frequency of occurrence, the variety in the choice of 
grammatical elements used to modify a main noun, and the level of elaboration 
through various linguistic processes such as embedding. For Dimension 2, adverbial 
expressions were examined using similar criteria. The frequency of occurrence, the 
variety of grammatical form choices that constitute adverbial expressions, and the 
level of lexical density achieved through adverbial expressions were considered for 
evaluation. For Dimension 3, references to characters and other text participants 
were evaluated in terms of the variety of positions where they appeared (e.g., in 
subject position, in object position, or in complement position). Whether the writer 
displayed a control of references to various characters was also used as a criterion 
for this dimension. The functional grammar rubric is found in Appendix D. 

The Scoring Session 

Preparation for an LAPA scoring session. A total of 1,646 student responses 
were evaluated during the 8-day LAPA scoring session. To minimize rater bias, all 
identifying information (i.e., students' names, student ID numbers, teachers' names, 
and school names) was removed from the student responses. Responses were 
numbered sequentially (i.e., 1 through 20), randomly distributed, and divided into 
84 packets containing 20 responses each. 

Thirteen raters (seven CRESST researchers and six middle school teachers) 
participated in the scoring session. Half of the teachers had participated in the 
writing institute, representing each of the schools involved in the study. 

Holistic scoring. Following the same general format described in previous 
CRESST work (Boscardin et al., 2004), scoring sessions began with a description of 
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the context of the project and the purpose and history of the LAPA. Although the 
raters were all familiar with the LAPA prompt and holistic rubric, they were asked 
to review them thoroughly once again. Following the review of the writing task and 
rubric, the raters read and discussed the anchor papers that represent three 
performance levels (score 2, 3, and 4 papers 1 ), starting with Anchor 2. The discussion 
centered on qualities of performance described in the rubric, and special attention 
was given to identifying distinctions in the quality of performance between Anchor 
2 and Anchor 3, and between Anchor 3 and Anchor 4. 

Once anchor papers were discussed, and before raters began scoring student 
work, the raters were asked to practice applying the scoring criteria by individually 
scoring a set of six practice papers (i.e., student responses that had been previously 
assigned scores by expert raters). As was the case with the anchor papers, the 
purpose of this practice set was to achieve high rater agreement. After scoring the 
practice papers individually, the raters discussed their qualities in reference to the 
rubric and anchor papers. Any major discrepancies were discussed thoroughly. 

Analytic scoring. Analytic scoring was based on functional grammar concepts. 
Except for three teachers who had not participated in the writing institute, all the 
raters were already familiar with the functional grammar concepts that were used as 
a basis for the functional grammar rubric evaluation criteria. Nonetheless, major 
functional grammar concepts were presented again prior to a rater calibration 
procedure for analytical scoring similar to that of the holistic training described 
above. In this presentation, raters were provided with the functional grammar 
scoring guidelines developed specifically for the functional grammar scoring 
training. The functional grammar scoring guidelines explained each of the three 
domains of functional linguistic skills in great detail. In order to help the raters' 
judgment in measuring students' linguistic skills and knowledge, strong and weak 
examples of target linguistic items were provided. 

Subsequently, the functional grammar rubric was distributed to the raters and 
was explained once again in reference to the functional grammar scoring guidelines. 
Once the raters indicated their familiarity with the functional grammar rubric, they 
were provided with five sets (one set for each functional grammar dimension) of 
anchor papers selected by the CRESST researchers. Originally, each set of anchor 
papers included four student responses that represented each score point (i.e., 1 

1 Since each anchor paper represents the lowest possible performance level for a given score point, an 
anchor paper reflecting a score of 1 is not needed (see Boscardin et al., 2003). 
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through 4 for each functional grammar domain). After initial evaluation of the 
scoring rubric, however, the measurement scales for each of the dimensions were 
reduced to 3-point scales, and anchor papers were adjusted accordingly. 

After 2 consecutive days of the LAPA scoring training, the raters scored 
student responses. The raters were provided with scoring sheets to record the six 
separate scores (i.e., one holistic score and five functional grammar scores) for each 
of the twenty responses in one packet. 

In order to ensure consistent reliability of the LAPA scores, a total of 9 "check" 
papers were scored each at different points during the LAPA scoring. If exact 
agreement fell below 70% on any of these check papers, it was discussed to 
determine the nature of the disagreement and recalibrated accordingly. 

When raters were finished scoring their first or second packet, we asked them 
to score the "reliability set" that comprised 30 randomly selected student papers in 
order to conduct a generalizability study on the scores. 

During the 8-day scoring session, there was slight attrition in the initial group 
of raters, and Dimensions 4 (evaluation, analysis) and 5 (impersonal structure) were 
excluded from evaluation for logistical reasons (e.g., time constraints) as well as 
lower agreement rates on these dimensions. 

Reliability of LAPA Scores 

In order to check for reliability of the LAPA scores, we conducted a 
generalizability study (G-study) with all trained raters on 30 randomly selected 
student assessments for each of the four rubric dimensions that were used. G-studies 
provided a closer look at the different sources of error (variability) in the scores and 
their relative importance, as well as the overall reliability of the scores. To examine 
the reliability of LAPA scores, we investigated specifically the amount of variability 
in scoring due to raters. The results from the G-studies suggest that the rater 
reliability and the overall LAPA scores were generally high. 

As shown in Table 6, for holistic scores, raters accounted for only 3.5% of the 
variability in the scores. Also, the variability in the scores due to raters was also very 
low for both Dimension 1 and Dimension 2 (7.6% and 4.1%, respectively). Although 
the variability due to raters was relatively higher for Dimension 3, it was still within 
the acceptable range. 
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Table 6 

G-study Results for LAPA Scores 





Holistic 


Dimension 1: 
Noun phrases 


Dimension 2: 
Adverbial 
expressions 


Dimension 3: 
Tracking 
participants 


Var (rater) 


0.02 (3.5%) 


0.03 (7.6%) 


0.01 (4.1%) 


0.06 (17.2%) 


Var (paper) 


0.27 (50.4%) 


0.18 (46.1%) 


0.11 (33.5%) 


0.10 (28.7%) 


Var (paper'rater) 


0.25 (46.0%) 


0.18 (46.3%) 


0.20 (62.3%) 


0.18 (54.1%) 



Teacher Opportunity to Learn Survey 

Teachers who administered the LAPA assessment also completed a teacher 
survey intended to capture critical aspects of OTL. The survey (found in Appendix 
E) contains five sections that include: (a) content exposure, academic language 
coverage; (b) content exposure, ELA content coverage; (c) access & development, 
ELL process strategies; (d) feedback & assessment; and (e) teacher experience and 
expertise. A brief description of these five areas follows. 

Content exposure: Academic language coverage. Within the survey, academic 
language was operationalized as instruction in the grammatical features needed to 
realize particular school-based writing genres. There were six specific items teachers 
responded to that measured students' content exposure to academic language. 
Teachers were asked to rate the average frequency in which they provided explicit 
instruction in various academic language topics in order to encourage students' 
writing development. Each topic represented a grammatical feature related to one of 
the three metalinguistic functions, field, tenor, or mode. Example items in which 
teachers rated frequency of coverage include: "verb choices that signal analysis of a 
character or situation" and "grammatical structures that generate an impersonal 
tone." Teachers responded on a six-point scale, ranging from "Never" to "2 or more 
times per day." These items are directly aligned the with assessment. Elowever, 
they were narrower in scope and targeted aspects of academic language described in 
the literature review section (e.g., long noun phrases and verb choices). The items 
also mirrored those included in the set of items targeting teacher content expertise. 

Content exposure: ELA content coverage. This variable was operationalized 
with nine questions that solicited information on the amount of class time spent 
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learning, or doing activities, related to literary analysis in both oral and written 
modes. Teachers reported time spent on these topics on a six-point Likert scale, 
ranging from "none at all" to "4 or more weeks" spent on each topic. Topics ranged 
from "summarizing the plot of novels, plays, or short stories" to "writing about 
heroic qualities of characters, sacrifices they make, or how they are courageous." 
These questions relate to topics directly targeted by the assessment and are 
consistent with the items associated with teacher content expertise. 

Access & development: ELL process strategies. The survey addressed 
variables that comprise access and development in a manner that conforms closely 
to the Sheltered Instruction Observational Protocol (SIOP) with its focus on 
sheltered instruction (e.g., use of supplementary materials, linguistic adaptation of 
content, etc.). The items were developed to reflect the general areas of the SIOP as 
described in the literature section. Due to space constraints, ELL process strategies 
and second language acquisition (SLA) strategies were collapsed into one variable 
for analysis. Questions targeting this variable centered on how frequently teachers 
used strategies to make the curriculum accessible to ELL students and how 
frequently they provided students with opportunities for extended discourse within 
various grouping configurations. For curriculum accessibility, the survey asked 
teachers how often they used supplementary materials, adapted content, linked 
concepts to students' background knowledge, adapted teacher talk, and scaffolded 
instruction. To gather information on how frequently teachers provided 
opportunities for students to engage in extended discourse, the survey asked about 
student-to-teacher interaction, student-to-student interaction, and opportunities to 
clarify ideas in the primary language. Due to space constraints and small teacher 
sample size, delivery format was not directly captured by the teacher survey. 

Feedback & assessment. This construct was operationalized in terms of teacher 
use of classroom-assessment data as well as the type of feedback teachers provided 
their students. In terms of assessment data use, four items asked teachers to report 
on the frequency with which writing assessments were used to assess students' 
understanding of literary elements, grammar, vocabulary, organizational skills, and 
spelling /punctuation. In terms of feedback, four items asked teachers to report on 
the frequency with which they provided feedback to students on their writing 
performance in relation to academic language concepts. Teachers were asked to 
respond to both sets of questions on six-point scales with 1 representing "never" to 6 
representing "almost everyday." Examples of items include: "Provide feedback to 
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students on their understanding of the role of paragraphs or sentences to support 
the writer's purpose (e.g., provide background, detail, and analysis)" and "Provide 
feedback to students on their understanding of vocabulary (e.g., verbs, adverbs, and 
adjectives) that reveals analysis of characters or situations." Due to the lack of 
significance this construct obtained in previous CRESST work, these items were 
modified to reflect the specific writing skills students need in order to do well on the 
assessment. 

Teacher experience and expertise. The survey measured three dimensions 
within this variable, including (a) preparation and training, (b) number of years 
teaching, and (c) content knowledge. Teachers' levels of preparation in the course 
content and pedagogy was operationalized in the survey in terms of the number of 
college level English language arts courses they had previously completed, and 
whether or not they received a graduate degree. Teachers were asked to report the 
number of courses completed that were directly related to the content of the 
assessment. They were also asked to indicate if they had completed a master's 
degree and to state specifically in what area this degree was granted. The idea here 
was that teachers with a master's degree in English literature would be better 
prepared to teach language arts than teachers without a master's degree or one in an 
unrelated field. To obtain a more complete picture of the degree of training, teachers 
were also asked to list recent professional development related to language arts. In 
addition, to capture the second experience variable, teachers were asked to report on 
the number of years they had been teaching. 

The third set of questions was related to the level of teacher expertise in 
content topics specifically targeted by the assessment, specifically knowledge 
involved in literary analysis. These questions consist of two categories: expertise in 
content typical of standards-based English language arts curriculum and expertise 
in content tailored for ELLs. As featured in previous CRESST work (Aguirre-Munoz, 
Kim-Boscardin, & Herman, 2002; Boscardin et al., 2004), general content expertise 
items included teachers' self-reports of knowledge related to literary elements such 
as theme and characterization. Content knowledge needed to support the learning 
needs of ELLs included knowledge of linguistic elements that comprise the kind of 
writing targeted by the assessment, response to literature. Since the definition of 
academic language knowledge in this study is based on functional grammar 
concepts, these items reflect this perspective. Teachers were asked to rate their level 
of expertise on 10 different items as either "novice," "adequate," or "expert." 
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Reliability of Teacher OTL Survey 

To ensure technical quality of the items on the teacher survey, we examined 
the reliability and the construct validity. The reliability of the items was evaluated 
using an internal consistency measure. Internal consistency measures are indicators 
of how well the items for each construct relate to each other. For a good measure of 
internal consistency, the alpha coefficient should be fairly high (e.g., > 0.80). The 
items on teacher experience were excluded from the analysis since the responses 
were directly translated into number of years. 

As shown in Table 7, the Cronbach alphas obtained for the five constructs 
(excluding the year of teaching experience) ranged from 0.86 to 0.95. These alphas 
provide evidence of high reliability across the five constructs. 

Table 7 



Reliability Coefficients for Teacher OTL Survey Subscales 



Content 


N 


Item no. 


Alpha 


Teacher content expertise 


29 


9a, 9b, 9c, 9d, 9e, 9f, 9g, 9h, 9i, 9j 


0.95 


Content exposure: ELA content 
coverage 


32 


10a, 10b, 10c, lOd, lOe, lOf, lOg, lOh, 101 


0.93 


Content exposure : Academic 
language 


28 


11a, lib, 11c, lid, Ilf, llg 


0.89 


Feedback and assessment 


28 


12a, 12b, 12c, 12d, 12e, 13f 


0.93 


ELL process strategies 


30 


13a, 13b, 13c, 13d, 13e, 13f, 13g, 13h, 13i 


0.86 



To examine the construct validity of the survey items, we conducted 
confirmatory factor analysis (CFA). Factor analysis is most appropriate for 
determining whether the items on the survey adequately captured the dimensions 
targeted. Nunnally (1978) suggested that "factor analysis is intimately involved with 
questions of validity... Factor analysis is at the heart of the measurement of 
psychological constructs" (pp. 112-113). 
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Table 8 

Confirmatory Factor Analysis for Teacher Survey 



Content 



Teacher content expertise 



Content exposure: ELA content 
coverage 



Content exposure : Academic 
language 



Construct 


Item No. 


Estimates 


S.E. 


Est./S.E. 


(a = .95) 


9a 


1.00 


0.00 


0.00 


(N = 29) 


9b 


0.93 


0.04 


23.69 




9c 


1.12 


0.07 


16.46 




9d 


0.98 


0.10 


9.98 




9e 


0.84 


0.07 


11.76 




9f 


0.63 


0.07 


9.63 




9g 


0.89 


0.09 


9.91 




9h 


0.77 


0.09 


8.28 




9i 


0.96 


0.07 


14.17 




9j 


1.08 


0.07 


15.94 


Reading 


10a 


1.00 


0.00 


0.00 


(a = .93) 


10b 


1.25 


0.14 


9.04 


(N = 32) 


10c 


1.22 


0.14 


8.79 




lOd 


1.37 


0.15 


9.03 




lOe 


1.38 


0.17 


8.23 




lOf 


1.34 


0.16 


8.41 




lOg 


1.18 


0.13 


8.97 




lOh 


1.30 


0.15 


8.47 




lOi 


1.33 


0.14 


9.25 


(a = .89) 


11a 


1.00 


0.00 


0.00 


(N = 28) 


lib 


1.05 


0.19 


5.58 




11c 


1.17 


0.21 


5.55 




lid 


0.52 


0.15 


3.53 




lie 


1.18 


0.17 


7.06 




Ilf 


1.60 


0.28 


5.70 
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Table 8 (continued) 

Confirmatory Factor Analysis for Teacher Survey 



Content 


Construct 


Item No. 


Estimates 


S.E. 


Est./S.E. 






Hg 


1.07 


0.21 


5.19 


Feedback and assessment 


(a - .93) 


12a 


1.00 


0.00 


0.00 




(N = 28) 


12b 


0.97 


0.06 


17.33 






12c 


0.92 


0.07 


13.96 






12d 


0.73 


0.07 


9.77 






12e 


0.83 


0.10 


8.21 






12f 


0.80 


0.10 


8.36 


ELL process strategies 


(a = .86) 


13a 


1.00 


0.00 


0.00 




(N = 30) 


13b 


0.52 


0.13 


4.06 






13c 


0.78 


0.12 


6.30 






13d 


0.57 


0.14 


4.09 






13e 


0.94 


0.11 


8.28 






13f 


0.83 


0.13 


6.16 






13g 


1.06 


0.12 


8.64 






13h 


0.91 


0.13 


7.14 






13i 


0.69 


0.14 


4.83 



Traditionally, exploratory factor analysis (EFA) is used to determine the 
number of factors or constructs that best describe the relationships among the items. 
However, the purpose of EFA is usually for theory generating rather than to test the 
quality of the proposed structures of the items. Confirmatory factor analysis (CFA), 
on the other hand, is traditionally used to test whether the items are sufficiently 
representing the specified constructs or content domains. 

As shown in Table 8, the results of the CFA suggest that the items in general 
seem to be adequately measuring the proposed constructs. The good model fit 
indices and high factor loading all indicate that the items have high internal 
consistency and factorial validity of the constructs. 
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Teacher Observation Protocol 

In addition to administering the teacher surveys, CRESST researchers observed 
classrooms of participating teachers at two different time points to gather OTL data. 
Initially, a total of 21 teachers were observed in classrooms approximately 2 months 
after the writing institute (Time 1). The same group of teachers was observed again 
during the LAPA preparation period (Time 2) along with six additional 
teachers. Teachers were observed for two consecutive lessons. These lessons were 
focused on elements that corresponded to the OTL instrument as well as more in- 
depth information about teacher practice. 

Observations of instructional practice were conducted using the Sheltered 
Instruction Observation Protocol (SIOP) developed by Jana Echevarria, and her 
colleagues (see Echevarria, Vogt, & Short [2000] for validation information). This 
tool was designed to capture the extent to which teachers' instructional practices 
reflect the features of effective sheltered instruction such as comprehensible input, 
building background, and metacognitive strategies (see Echevarria et al. for a 
detailed description). The observation protocol was also adapted to meet the needs 
of the current study by reflecting the new ELL-sensitive OTL model with inclusion 
of items related to academic language instruction (see Appendix F). 

Content exposure: Academic language coverage. Two questions on the 
observation protocol were utilized to address this variable, both directly focused on 
students' level of exposure to functional grammar in classrooms. The first item 
asked observers to rate the degree to which student activities facilitated the 
application of language concepts in the classroom generally, and the second item 
asked observers to rate teachers' instruction of specific grammatical features. For the 
first item, observers rated teacher's provision of language related activities on a scale 
from "highly evident", meaning the teacher provided activities for students to apply 
language knowledge in the classroom, to "somewhat evident," where the teacher 
provided activities for students to apply either content or language knowledge in 
the classroom, to finally, "not evident," where the teacher provided no activities for 
students to apply content or language knowledge in the classroom. The second 
question that addressed this variable included six sub-questions. For each of the six 
sub-questions, the observers rated teachers on whether particular functional 
grammar principals were addressed in the classroom during the observed lesson to 
"a great extent," "some," or "barely covered." These functional grammar principals 
include (a) use of various clauses and phrases in theme position to create sentence 
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variety; (b) use of adjectives, verbs, and adverbs to reveal the writer's evaluation; (c) 
use of various verb types to provide textual interpretations; (d) use of connectors 
and expanded noun phrases to build cohesion; (e) use of model verbs that frame the 
writer's point of view; and (f) use of grammatical features such as expanded noun 
phrases to generate an impersonal context. 

Content exposure: ELA content coverage. This variable was not directly 
measured in the observation protocol, as it focused mainly on the variables related 
to Access and Development, and Academic Language. Yet observers did regularly 
witness students being exposed to ELA content topics, as indicated by their field 
notes. These notes have provided additional information for the analysis of patterns 
in teacher practice. From these records we have been able to closely examine 
learning opportunities occurring in the classrooms, as well as the nature of the 
literary analysis, reading, and writing activities in which students participated. 

Access and development. In the observation protocol, seven items were 
included for qualitative analysis (described in section titled "Capturing Opportunity 
to Learn In Teacher Practice") reflecting various teacher practices related to this 
dimension. Five of these items targeted SLA strategies, namely different forms of 
comprehensible input; one item targeted delivery format (grouping configurations), 
and the final item related to ELL Process strategies (scaffolded instruction). 
Observers rated teacher practices on a 5-point Likert scale, ranging, for example, 
from 1 representing "speech inappropriate for students' proficiency level" to 5 
representing "speech is appropriate for students' proficiency level (e.g., slower rate 
and enunciation and simple sentence structure for beginners)." 

Feedback and assessment. This dimension of instruction was addressed by 
judging the quantity and quality of teacher assessment and feedback to students 
based on students' oral output. Judgments were reported on a scale ranging from 0 
indicting that the teacher "provides no meaningful, specific feedback to students on 
their output" to 4 indicating that the teacher "regularly provides meaningful, 
specific feedback (e.g., language, content, or work)" to students. For assessment 
practices, judgments were also reported on a scale ranging from 0 indicating that the 
teacher "conducts no assessment of student comprehension and learning of lesson 
objectives" to 4 indicating that the teacher "conducts assessment of student 
comprehension and learning of all lesson objectives (e.g., spot checking, group 
response) throughout the lesson." 
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Data collection. This protocol allowed the research team to focus attention on 
gathering evidence of effective sheltered instruction and simultaneously collect 
information on the level of exposure to academic language. 

Additionally, observation field notes were taken to obtain a more complete 
picture of classroom processes to contextualize information gleaned from the 
analysis of the SIOP and teacher interview data. As discussed below, the results of 
observation data (i.e., SIOP data and field notes) analysis, survey data, and teacher 
interview findings were triangulated in order to identify the OTL factors that impact 
ELLs' academic achievement. 

To achieve high inter-reliability amongst researchers, the research team 
participated in an observer calibration session prior to site visits that involved the 
use of video-taped lessons, discussions around key constructs and practice 
observations at one of the pilot-testing sites. Further, during site visits for the study, 
a pair of observers teamed up, observed a teacher, and filled out a SIOP form 
together after a thorough discussion of the observed classroom practices. 

For quantitative purposes, the items in the observation protocol were grouped 
into two larger inclusive constructs called "teacher's effectiveness at providing clear 
content expectations," and "engagement in enticement activities." These constructs 
were then analyzed to determine their relationship to student outcomes. The 
findings of the quantitative analysis are described in the section "Results from 
ITEM." 

Teacher Interview Protocol 

Interviews with participating teachers were conducted using a modified 
version of a semi-structured interview protocol previously developed by CRESST 
(Griffin, Aguirre-Munoz, Goldschmidt, Amabisca, Miyoshi, Swigert, & Trusela, 
2003). The modified version was designed to correspond to the ELL-sensitive OTL 
model by gathering information on the background experiences of teachers, the 
implementation status of academic language instructional strategies, use of ELL- 
specific classroom processes, and other information relevant to the current study 
(see Appendix G). 

For the qualitative analysis, which focuses on patterns of teacher practice and 
levels of classroom OTL, we reviewed teacher responses to particular items from the 
interview protocol and analyzed them in line with the various constructs. This 
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operationalization of the OTL constructs in the interview protocol is further 
described below. 

Content exposure: Academic language coverage. The interview protocol 
addressed this variable with questions targeted at academic language instruction, 
and for trained teachers, additional questions about the classroom application of 
CRESST writing institute concepts. These included questions such as, "How do you 
prepare ELLs to process academic language?" and "Which activities/ strategies have 
you used from the institute so far?" Researchers also reviewed teacher responses to 
questions pertaining to the observed lessons, which had often led to a discussion of 
typical classroom practices, including whether or not they regularly included 
academic language instruction in their practice. Answers were then coded for 
specific grammatical features or functions, including: "grammatical structures that 
build cohesion," "vocabulary that reveals interpretation," "long noun phrases to 
increase sentence variety," and "overall essay cohesion and organization." 

Content exposure: ELA content coverage. ELA content coverage is 
operationalized in the interview protocol and transcript-coding schema as 
instructional content related to literary analysis, general reading activities, the 
writing process, and general writing activities that did not directly address academic 
language from a functional grammar perspective. Activities and instruction related 
to academic language as defined above was coded instead for academic language 
coverage. While questions did not target ELA content coverage directly, in the 
interview protocol the questions related to teachers' descriptions of observed lessons 
provided the most information on ELA content coverage. Teachers were asked how 
the lessons fit into the development of the unit in which they were situated, and the 
overall purpose of the instructional unit. In this way, researchers were able to detect 
the types of topics covered in the classrooms. A few questions from the ELL-specific 
constructs in the interview protocol also provided information on ELA content 
coverage. Examples of these questions include: "Give an example of how you have 
your ELL students carry out oral and written tasks that require the following: 
comparison/contrast, description, explanation, and definition" and "Which specific 
strategies have been most successful with low proficiency ELLs and moderate 
proficiency ELLs?" While this second question targets instructional strategies as 
opposed to content, many teachers described the types of content they covered in 
order to improve the achievement of ELLs in English language arts. Teacher 
responses about classroom content coverage were coded to indicate whether the 
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content was related to reading or writing. These categories were then further 
divided into sub-content areas such as reading comprehension, literary analysis, 
reading practice, pre-writing, revising, and writing practice. 

Access & development. In the interview protocol, there were four questions 
specifically targeting teacher practices intended to support students' second 
language acquisition, in particular, students' opportunities for extended discourse. 
These questions asked teachers to describe the types of opportunities for extended 
discourse they provided students, the type of circumstances in which these 
opportunities arose, and their frequency , including frequency of pair and group 
work. Additionally, the interview protocol included questions that solicited 
information on teachers' delivery format by prompting them to describe the last 
time their students worked together within a small group setting. Teacher responses 
to these items provided information on the levels and quality of this OTL variable. 
Teachers' descriptions were coded to indicate the instructional format (student 
grouping) and the specific type of strategies used that were relevant to the access 
and development variable, such as comprehensible input, graphic organizers, and 
link to student background. 

Feedback & assessment. The interview protocol focused on how teachers 
prepared students for the LAPA assessment with items specifically related to LAPA 
preparation. These questions targeted teacher practices in relation to OTL by asking 
them to report on their use of instructional modifications to accommodate ELL 
learning needs (process strategies), and to report on student content exposure by 
asking how students were prepared for the LAPA (content coverage). Additionally, 
questions soliciting information on the lessons observed during the LAPA writing 
period were informative for this construct, as they provided both detailed 
information on particular lessons, and general patterns of classroom practice during 
the assessment period. 

Teacher experience and expertise. To check for consistency with the survey 
responses, teachers were asked specifically how many years they had taught, 
whether they had a credential, and if so, what type (e.g., BCLAD or CLAD). 
Teachers' answers were coded with their credential status and credential type. 
Correspondingly, teachers' responses indicating their number of years teaching 
were coded into five categories including: less than 1 year, 1 to 2 years, 3 to 5 years, 6 
to 10 years, and 11 or more years. Since the purpose of the interview was to obtain 
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more detailed information regarding teacher practices, information about teacher 
content expertise was not addressed in the interview. 

Teacher Interview Data Collection 

Each teacher was interviewed during each of the two classroom observation 
time points (Time 1 and Time 2). The audio-taped interviews were then transcribed 
and coded according to a coding scheme developed primarily based on response 
patterns in teachers' ELL-specific instructional strategies, LAPA preparation, the 
writing institute, and functional grammar concepts. Table 9 presents a sample of 
codes used for analyzing the teacher interviews. 

In order to maintain coding consistency, each of two researchers 
independently coded a set of interview transcripts, reviewed each other's 
preliminary codes, and refined them further in consultation with each other. This 
process of code refining continued until researchers reached a 76% agreement rate. 
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Table 9 



Sample of Codes Used for Teacher Interview Analysis 



Code 


Description 


Example 


Teacher expertise 


Comments regarding teachers' 
credential status and teaching 
experience. 


Q: What's your credential status, what 
kind of certification do you have? R: I have 
a life credential. Q: Do you have a CLAD 
or a BCLAD? R: I have a CLAD. 


ELL population 


Comments regarding the size of the 
ELL population in the classes. 


Q: How many English language learners 
do you teach? R: I probably have about 10 
to 12 in different classes; they are not all in 
one place. 


Activities - Reading 


Comments regarding instructional 
content and delivery pertaining to 
reading such as literary analysis and 
vocabulary building prior to reading. 


In reading, I can get them excited about 
what we're into. I started off initially with 
short stories out of the literature book. We 
also have read The Diary of Anne Frank. We 
have prescribed things we have to teach. 


Activities - Writing 


Comments regarding instructional 
content and delivery pertaining to 
writing such as response to literature, 
writing process, writing conventions, 
etc. 


Basically, we started the year out just by 
doing a lot of free writing like unstructured 
writing to get them used to thinking and 
getting their thoughts on paper . . . .When 
we do our writing between our first and 
our final draft they also do peer editing 
where they have to read each others' 






papers. 


Instructional formats 


Comments regarding different 
instructional formats such as 
individual, small, or whole group. 


They were supposed to read together, 
discuss what the plot was, and then they 
each had to do a graphic organizer. I think 
I got about on average three per class. They 
drift into other things. They just don't stay 
on task. 


ELL strategies 


Comments regarding ELL-specific 
strategies. Statements about various 
scaffolding techniques such as using 
graphic organizers and visual aids. 


If the students are not getting it then we 
stop or switch gears or change direction, 
get more graphic, point to the picture in 
the book and say it again or whatever it 
takes. 


Institute functional 
grammar 


Comments regarding explicit 
instruction of academic language, such 
as teaching functional grammar 
concepts. 


For our second paper, we might talk about 
adding adjectives. So after they do their 
rough draft they'll underline or highlight 
every noun in their paper, and then we 
brainstorm different adjectives and things 
like that on the board. 


LAPA preparation 


Comments regarding how ELLs were 
prepared for the LAPA, including 
statements about content areas covered 
and specific strategies used for the 
LAPA preparation such as teaching 
functional grammar, going over the 
writing prompt and rubrics, etc. 


I began with an explanation of what they 
were going to be doing, and what the 
expected outcome was going to be. I 
explained the prompt, so they could 
understand exactly what they needed to 
do, and went over suggestions on how 
they can keep it together, and mini-lessons 
on prepositions and phrases that helped 
them to develop their paragraphs. 
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Teacher Rating 

We used the interview data to develop two OTL variables related to coverage 
of academic language and inclusion of ELL process strategies for the quantitative 
analysis described in the section entitled "Results from HLM". Specifically, we rated 
teachers on their instructional practices in order to gauge students' exposure to 
academic language content coverage and ELL process strategies during their 
preparation for the LAPA. The decision to use the second observation time point 
was to increase our sample for the quantitative analysis. Interviewed teachers were 
rated on two measures. On each measure, teachers were given a score of 0, 1, 2, or 3. 

Lirst, we examined the extent and quality of teachers' academic language 
instruction, operationalized as functional grammar implementation. Second, the 
extent and quality of ELL process strategies was evaluated. The ratings were based 
on teacher responses to interview questions targeting how they prepared their 
students for the LAPA. To determine teacher ratings, researchers developed two 
rubrics (one for functional grammar and another for ELL process strategies), which 
were reviewed by the research team for coherence and consistency (Appendix H 

and Appendix I, respectively). Two researchers then assigned each teacher a score 
on each of the two measures by reading through interview responses queried for 
"LAPA Preparation," as well as by reading through Time 2 interview documents 
when the information in the query appeared incomplete. One of the researchers had 
previously viewed the students' averaged LAPA scores before the rating process, 
but did not consult this information while determining teacher ratings. The second 
researcher did not have any exposure to student LAPA scores prior to determining 
teacher ratings. The reliability between the two researchers on the functional 
grammar ratings was high at 86%, while the reliability for the ELL process strategies 
was lower at 54%. Here, reliability is defined as exact agreement between the two 
researchers. The lower reliability for the ELL process strategies was likely due to the 
fact that many classroom factors impacted the type of ELL process strategies 
appropriate for the different classrooms. As the raters had different initial 
knowledge of these circumstantial factors (e.g., number of ELLs in a given classroom 
and the range of student ELD levels effecting the extent to which teachers 
appropriately differentiated instruction), they at times rated teachers' ELL processes 
higher or lower than the other. After discussing the rating differences, the raters 
reached consensus on all of the academic language and ELL process strategy ratings. 
The current ratings represent this consensus. 
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Teachers were rated on two dimensions, depth and breadth, for both functional 
grammar and ELL process strategies. Depth refers to the level of accuracy and 
complexity in the specific ELL process strategies and the functional grammar 
concepts targeted for instruction, determined by the level of detail in teacher's 
descriptions of these practices. Additionally, depth refers to these concepts' level of 
integration into writing activities and expectations. Lor example, when determining 
the level of functional grammar implementation in a description of instruction 
targeted at expanded noun phrases, we considered whether instruction focused only 
in adding adjectives to modify nouns, or whether instruction also included other 
strategies for noun expansion such as the use of prepositional phrases and 
embedded clauses. Lor each functional grammar concept or ELL process strategy, 
we also examined the extent to which a teacher identified the particular problems in 
student writing that the strategy was meant to remedy. 

The dimension of breadth refers to whether a variety of ELL process strategies 
and functional grammar concepts were targeted in instruction as well as the 
duration of such instruction. We considered whether the concepts were taught in a 
progressive nature (if concepts built upon one another) to expand the students' 
knowledge and skills of academic writing. We also looked at whether or not 
functional grammar was an important aspect of LAPA preparation overall and the 
extent to which teachers utilized ELL process strategies in this instruction. Ligure 2 
presents four example teacher profiles that illustrate each of the rating levels. 
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Rating 3 


The teacher: 

• Taught functional grammar systematically over the 4-month period between CRESST training and 
administration of the LAPA. 

• These features included nominalization, participial and prepositional phrases, subordinate and 
embedded clauses, expanded noun phrases, process (verb) variety, compound and complex 
sentences, and marked themes (the use of contextual and linking information before the subject). 

• Reviewed grammatical features during 2-week LAPA writing period. 

• Utilized systematic instructional procedure for teaching each targeted grammatical feature which 
was comprised of: 

• Introduction to grammatical feature with explanation of its function in text, ideal patterns for 
usage, and common problems in student writing; 

• Opportunities for students to practice identifying and using grammatical feature using a variety of 
techniques; 

• Opportunities for students to practice using grammatical features in the creation of lengthier 
narrative and expository texts using the writing process, with a focus on the revision stage. 

• Made functional grammar concepts central to LAPA preparation instruction. 


Rating 2 


The teacher: 

• Implemented functional grammar instruction at various time points over the 4-month time period, 
covering a breadth of topics but not prioritizing class time writing practice. 

• Features included theme/rheme, expanded noun phrases, prepositional phrases, and process (verb) 
variety. 

• Grammatical features were reviewed during the 2-week LAPA writing period 

• Included a variety of functional grammar instructional processes including guided and individual 
identification of grammatical features in text, and the revision of text using targeted grammatical 
features. 

• Provided students with visual aids such as theme/rheme charts and word walls. 

• Provided students with opportunities to use grammatical features in writing paragraphs and extended 
text, instructing them to revise text for targeted grammatical features; the approaches are limited. 

• Provided verbal and written feedback to student writing focused on targeted grammatical features 
(e.g., expanded noun phrases and verbs). 

• Made functional grammar an important aspect of LAPA preparation instruction. 


Rating 1 


The teacher: 

• Provided little functional grammar instruction. 

• Functional grammar was the subject of one observed lesson during the 2-week LAPA writing 
period, which included transitional phrases to elicit different relationships between ideas. 

• Did not highlight functional grammar in LAPA preparation. 

• Focused writing instruction during the LAPA writing period and the 4 months prior on writing 
content and not language or grammatical structures. 

• Focused primarily on reading comprehension during writing instruction. 

• Provided students with (unspecified) writing structures during writing assignments. 

• Made functional grammar a minor aspect of LAPA preparation instruction. 

The teacher: 

• Provided no functional grammar instruction during the LAPA preparation period or previously. 

• Language arts instruction focused on reading comprehension strategies and superficial aspects of 
writing, such as global essay structure and mechanics. 


Rating 0 



Figure 2. Example profiles of functional grammar instructional practices for each rating. 
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Analysis 



As mentioned earlier, we used a mixed-methods approach to examine the 
research questions in this study. The mixed-methods design is described as research 
that incorporates both quantitative and qualitative data in a single study (Creswell, 
2003). The mixed-methods approach is based on the theory that no single method 
adequately provides enough information to address the research problem and in 
this way each method reveals certain aspects of empirical reality (Denzin, 1978). 
Grounded in this theory, we have attempted to triangulate and incorporate all data 
sources into our statistical analyses as well as examine each data source separately. 
In this section, we'll briefly describe how the different data sources were utilized in 
our statistical analysis to address our research questions. In addition, we'll describe 
the methods used for analyzing the interview and classroom observation data. 

Ordinal Logistic Hierarchical Linear Models 

Responses to the OTL survey, interview data, and classroom observation data 
were analyzed in concert with student performance results using two-level ordinal 
logistic hierarchical linear models (ordinal logistic HLM). The factors influencing 
student performance occurred in the context of classrooms, which gave rise to 
multilevel data. Usually, students within the same classroom are affected by similar 
factors such as the teacher characteristics, educational resources, as well as the 
environment of the classroom. HLM models provided a systematic way to 
investigate how teachers, and specifically the OTL variables, influenced student 
outcomes and whether these variables have any differential impacts on ELL 
performance after adjusting for student-level variables. Given the four-point scale of 
the LAPA holistic score and the three-point scale of the three dimensions, we 
examined the relationship between LAPA scores and classroom differences 
characterized by OTL variables using ordinal logistic HLM. The final HLM model 
specified in our study is as follows: 

p m : Prob. (outcome category =m) 

p* m : Prob. (outcome category < m) = p x + p 2 b p m (therefore, p* 4 = 1) 

^Category 1 represents the probability of getting the highest LAPA score (4 for the Holistic 
score and 3 for Dimensions 1-3). 
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